version 17.08

This release adds support for odk parameters.   Actually currently there is only one parameter supported and that is max-pixels which can be used to reduce the size of photographs taken in a form.

Usage

Open the form in the online editor and select “Parameters” as the property you want to change.  Then find an image you want to reduce.  Then set the max-pixels parameter to the size you want.  For example:

max-pixels=500

The number is the maximum length in pixels of the long edge of the image.

Notes

  1. As well as SmapServer 17.08 you will need fieldTask version 5.06
  2. There is no support in xlsForm for parameters yet hence you will need to enter the parameter in the online editor as described above.
  3. You can consider parameters and appearance as doing very similar things .  In the past appearance was used for many of the capabilities that odk will now be adding using parameters.
  4. Help is available in the reference guide.  https://docs.google.com/file/d/0B5_SmpWlQYxvWlBoWkdiT3FrY1U/edit

 

version 17.07

Automatic Phone Refresh

When there is a change to a survey, or the data that a survey references, then these changes can be pushed automatically down to all the affected phones.

Changes automatically refreshed include:

  • New survey added to a project (See video)
  • Survey Edited (See video)
  • Survey deleted
  • Survey Blocked
  • Update to a dependent survey. For example if a survey gets reference data from the results of another survey then updates to those results will result in the phone being refreshed (See video)
  • Change in phone users settings, ie added to a new project or new security settings
  • Tasks: added, deleted, assigned, unassigned, properties changed

You can specify automatic refresh in the organisation settings or in the phone settings for Wifi, Cellular or both. The organisation settings, if set on, will override the settings on the phone. Note these settings have been used for a while but previously would only automatically submit finished results to the server. Now they also automatically download new versions of the form.

Timing Data

The latest version of odkCollect and FieldTask 5 can collect the time taken to complete each questions.  You can now enable forms for this timing data and then view this data through the API.

 

Other New Features

  • “Acknowledge” question type added to the online editor.  This creates a question with a single checkbox.
  • Updated the Kobo API to return data from sub forms as well as the main form.  Using this link https://sg.smap.com.au/api/v1/data (user: gplay, password: gplay) returns a list of the available forms.   For the form titled “Sub Forms” you will see there is an extra URL returned that allows you to select the sub form of data.  To support sub forms additional parameters have been added such as “start_parkey”.  For details refer to the documentation.
  • Upgraded FieldTask to include odkCollect 1.8.1.   This is considered a major release so it has been given the version number 5.0.  The reason being that this update no longer supports versions of Android up to 4.04.  Hence FT4 will be kept available for older versions.  FT5 will be in google play shortly and will then be progressively rolled out onto servers.

 

Version 17.06

The june release is now available.

Repeat in Webforms using repeat count

New Features

  • Webforms has been updated to use the latest version of enketo-core, some new capabilities include:
    • Images are included in the choices that can be filtered using choice filters or cascading selects
    • Using repeat counts on begin repeats is really cool.  Especially if you make the repeat count dependent on another question.  Repeats are shown without buttons to add or remove and the number of repeats will change as you change the repeat count.
    • Plus lots of other styling improvements
  • When you create  PDF from results using a template, if you specify a text field with the name of an image question then a hyper link will be added.  If you are not using a template then you can add the appearance “pdfhyperlink” to show the link instead of the image itself.

version 17.05

Here it is finally. There was no release of the Smap server in March or April.

New Features

  • Integer and decimal results exported to XLSX files are now in number format in the spreadsheet.  Dates are in date format.
  • A “display_name” parameter has been added to questions.  This allows you to override the question name with something more user friendly for exports.
  • The “Unique Key” that you can specify for a form is now included in PDF exports.  You can also set the key value to be a serial number starting from an offset. For example:
    • serial(100)  will create keys of 100, 101…
    • REFserial(10) will create keys of REF10, REF11…
  • The charts on Browse Results and Managed Forms have been substantially modified, (this is still beta code)
    • Maps can now be added
    • Charts are added by the user and then saved with the survey.  This is in contrast to the previous approach where charts were added automatically for all questions causing performance problems on large surveys.
  • When exporting data to PDFs, for location questions you can specify an appearance of pdfzoom_xx, where xx is a number between 1 and 21.  This will show a map of the data at the specified zoom level.  The higher the number the smaller the area covered by the map.
  • Notification emails can now include survey name, instance name, unique key, user and device anywhere in the subject or body of the email, use:
    • ${surveyname}
    • ${instancename}
    • ${hrk}
    • ${username}
    • ${device}
  • Email notifications.  If you set up a notification in managed forms then an email will be sent to the address of the user to be notified

Bug Fixes

  • Updating of user ident is now prevented.  If you tried to change your logon id in the past you would not have been able to logon.  This is because the logon id is combined with the password to create the real encrypted password.  The only way to change your user ident is to create a new user and delete the old one.

Smap at AWS Summit – Berlin 18th May 2017

Yes Berlin which I am very excited about, even more so as I am attending the AWS summit there.   This will be an opportunity to get inspiration for Smap 2.0, the second generation of Smap software. I will be updating this post regularly up until the end of the Summit to record some of my thoughts on where the Smap architecture is headed.

Smap 2.0

  • A new system that will work with the existing Mobile Data Collection Servers and Phones
    • Runs in cloud
    • React.js front ends
    • GraphQL services where possible
    • Serverless back ends
    • Single sign on across mobile devices and servers
  • Will be used to deliver notifications and process management capability to the existing servers
  • Manage notifications to and from mobile phones.  For example notifying a phone that a new version of a survey is available
  • Plus much more

AWS Summit

None of the following comments represent design decisions for Smap 2.0 they just take the concepts from a topic and construct some potential scenarios.

Summit Demo: MXNet on the edge.  How to build a deep learning model that works on edge devices

A Smap server could be considered an edge device.  The data collected could feed a machine learning application to provide real time insight into data and quality assurance.

The data processed by the machine learning could include the time taken to answer each question in order to  identify poorly complete surveys.  The collection of question timing data is currently being added to ODKCollect and fieldTask in a joint Smap / ODK project.

The demo at the AWS summit is going to use a Raspberry Pi as the edge device to recognise art objects.   A mobile phone running fieldTask could also be an edge collection device interacting with Smap 2.0.  This is how the phone will get notifications to refresh forms etc.

MXNet is a neural network application from Apache which can be trained in order to recognise patterns.

Summit Demo: Utilize Powerful Visual Search and Discovery Capabilities with Amazon Rekognition

Rekognition is one of several AI services provided by AWS

  • Rekognition: Image recognition
  • Polly.  Text to speech
    • Lex.  Speech and text recognition and language understanding

It looks like there are quite a few sessions on AI at the summit and adding AI capabilities to Smap is something we are considering so this could be useful.

Version 17.02

New Features

  1. Using the Browse Results and Managed Forms pages, you can now filter between a begin and end date.  Any date in the data set can be used for this filtering.  This allows reports to be created for specific data ranges such as a month.
  2. Export to XLS from Browse Results and Managed Forms now creates a worksheet for each chart.  The data in these worksheets can be selected to recreate the chart in the spreadsheet however you also have all the flexibility of the spreadsheet to style the chart the way that you want it.
  3. The filter setting used in an export to a spreadsheet for Browse Results and Managed Forms are now shown in a “settings” worksheet in the export.  Hence you can see if the data is for a specific date range or only those records that match the specified text etc.
  4. Cascading selects created from an external CSV file now work in web forms.
  5. Pulldata functions to look up values in a CSV file now work in web forms.  Thanks to Martijn van de Rijdt @ enketo.org.  (Update this feature has not ended up in 17.02 and will be delayed to 17.03)

Version 17.01

Version 17.01 is now available, it includes two new features:

  1. Editing of cascading selects and choice filters in the online editor
  2. Accessing records from another survey using pulldata

Choice Filters

You can now create and edit choice filters in the online editor.  This should be an important feature as even if you develop your form using the excel editor it is fairly common to want to add a few more choices to a cascading select after you have started collecting data.  The online editor allows you to do that as you can change the form without affecting the data already collected.

Cascading selects are a type of choice filter where the choices in a select_one question are determined by a selection made in a previous select_one question.  Because these are so common the editor provides extra support to simplify their use.

However if you need to do something more complex then you can use the custom filter option which has all the flexibility and power of available when you create choice filters in the excel editor.

The help page has more details.  You can also watch the following video which shows creation of both a cascading select and a custom filter.

Looking up data from another survey

Smap offers two functions that can retrieve data from CSV files:

  • search()  Returns choices for a select
  • pulldata() Looks up individual data items using a key

One of the features in Smap that has been there for a while is that you can get the system to automatically generate and update the CSV file from the data in another survey.  To do this specify the “file name” as “linked_” then the survey identifier that has the data.

However this was limited when it came to pull data:

  • the key could only be a single question in the survey that has the data
  • you couldn’t look up repeating data in the other survey, ie where there was more than one record that matched the key.

Both of these issues have now been addressed.  For example you may have a survey that has collected data on children registered at a school:

School   Class      Student Name

school class student
East A Fiona
West A Karen
West A Harry
West B Tom

If we then specify the key to look up this data as: ${school}_${class} and that it is a repeating key then the CSV file that is generated for lookups will look like this:

_data_key _count student
East_A 1
East_A_1 Fiona
West_A 2
West_A_1 Harry
West_A_2 Karen
West_B 1
West_B_1 Tom

The _data_key and _count columns are automatically created by the server.  The “student” column is there because there is a pulldata() function in the form that gets the answer to the student question.  Because there are many students in each class a row is created with a value for _count for each class.  Underneath that are rows for each student in the class.  The key for these students is the same as the class key with the addition of_1, _2 and so on.

To access this data from a form you can:

  1. Get the school and class from the user.
  2. Use these to look up the count for the number of students.
  3. Then in a repeat group add the value from position() to the school and class to get details for each student.

Details on how to do this are currently being updated in the training material.

Styling Forms with Color and Links

One of the new features available with fieldTask version 4 is the ability to add color and other styles to forms.  This feature was contributed to odk by Nafundi and you can find out more on their website.  https://nafundi.com/blog/posts/customize-odk-forms-with-fonts-colors-sizes-and-links/

If you want to try it out there is an example form, also created by Nafundi although I have made some minor changes to it.

Field Task Version 4

Field Task version 4 has finally been released.

Support for version 4.1.2 of ODK Collect

This adds a lot of great functionality provided by the ODK team.  Details on the new Collect functionality is available heres. https://github.com/opendatakit/opendatakit/wiki/Collect-Release-Notes.  Some of the main new features are:

  • Can select Google or OpenStreetMap maps as the base map for a geopoint question or the tasks map tab.
  • The Google base map can be chosen from; streets, terrain, satellite or hybrid
  • Custom map layers can be added.  These maps mbtiles format. They can then be copied into a folder under the “layers” folder in the fieldTask folder on the phone. Custom maps can be generated from floor plans of building or other images.

Task Map with Google Satellite Background

Using a custom map, created from the floor plan of a building, to mark out a polygon

Other features

Routing to tasks.  You can now select a task on the maps tab then select the Google Route button to create a route.

Version 16.12

Importing data from a file into a survey

  • Data can be imported into sub forms
  • Polygons and lines can be imported
  • Data can be imported from xlsx files as well as CSV files
  • A zip file can be exported containing
    • An xlsx file with all the data for all forms / sub  forms in the survey
    • Images, video, audio files from the survey
    • In a format that can be immediately imported into another survey or, after modification, back into the survey it was exported from
  • User, Deviceid, notes, start time, end time, original upload time are all exported and then imported into the new survey

The following video demonstrates how this now works:

Other Changes

Uniqueness of question names

Question names now need to be unique across the entire survey and not just within a single form. This is a point of difference between Smap and other ODK systems so you may find that some forms that work with a different ODK variant will break when uploading into Smap until you make the question names unique.

Option to remove ODK menus from fieldTask

Using fieldTask when you press the “refresh” button new forms that you have access to will be downloaded and results will be submitted.  Successfully submitted results will be deleted if you have the option set on the server to do this.  You can also start editing a form by selecting it on the task list page.  However the ODK menus to edit, delete, get forms and submit results are still available.

In the organisation settings you can now select an checkbox under fieldTask options to remove these menus simplifying the interface for the user.  This will also prevent the user from deleting un submitted results.

This feature requires fieldTask 14.02.

Option to prevent review of results on the phone

Previously a phone user could read the data collected in a survey and add comments after the survey was completed.  You can now set an option on the server in the organisation settings to prevent this.

This feature requires fieldTask 14.02.

Support for Ubuntu 16.04

If you are installing your own server the install script now works with Ubuntu 16.04. Ubuntu 14.04 is also still supported.