Harmen Kampinga is a senior 3D specialist and combines DATA, GIS, ET, 3D and Clearly in his daily work. He is a Certified FME® specialist with an eye for detail.
From Future Insight we explain in a triptych how we help our customers to get a grip on these projects with the powerful combination of Clearly and FME.
Bas wrote in the first part about the use of Clearly in Construction Teams. Rick then went into more detail on how we can make projects run a lot smoother with Clearly. These processes are fed with data. In this part I show how we use FME to integrate precisely that data in a smart way in order to make our approach better, more reliable and faster.
FME?
Why make it difficult when it can be done easily? That is exactly the reason to use FME from Safe Software. FME stands for 'Feature Manipulation Engine' and is, in short, a platform that is strong in the rapid processing of data and spatial data in particular. It is the tool for us to process all types of data effectively and efficiently and to put it on the map. Moreover, it helps us to guarantee consistent quality in our projects.
Rick already indicated in his previous blog that a lot of information is available for an average infrastructure project. Often, important information about the project is scattered across numerous documents and systems, and often in PDFs as well. As a result, much of the original traceability of information is lost. Bringing all this information together, linking it to the map and making it transparent for the Client and the contractor is what we do with Clearly. Previously much of this information was processed manually, time consuming and intensive.
It had to be done differently. And that is why we started using our first process in FME in 2017. That went very well and since then we have set up more and more processes in this way. This allows us to significantly shorten the lead time of processing data.
Data Sources
We use various data sources in our projects. We roughly distinguish two types of data:
open data, such as spatial plans, cadastral data, drilling and soundings;
project data, such as boundaries, KLIC data, construction locations and construction roads.
The open data sources have a wide variety of formats and options. Examples are the Basic Registration of Topography (BRT) and the Basic Registration of Large-Scale Topography (BRT) of the Land Registry. For this we use FME to make the datasets suitable for use in our projects. This way we cut out the project area from these datasets and select only the relevant attributes. We then supplement the dataset with metadata so that it is always clear what the source and topicality of the data is.
Data Sources
An example of the combination of various open data sources, such as Spatial Plans, Cadastral data and 3D BAG.
We see the real challenges in project data. As Rick also mentioned in his blog, it happens all too often that we receive the data in a tender as a ZIP full of PDF files. Think of a design, soundings or the project location. By asking questions until we get the source files, we sometimes get a long way, but just as often we don't. Source files are not present. The reason? The data cannot be traced back or a government does not want to make it available or does not have it ready in the request / tender. Missed opportunities that we see all too often….
Now we ensure that, one way or another, something useful is made of the 'data'. We process PDF files with, among other things, FME into readable datasets or split a report into smaller PDF files. This sometimes happens with (old) CPT data that were once delivered in a report (PDF). With the right steps, we can turn this into a points file. Each probe then receives the relevant data plus a reference to a drawing from the relevant document. A big job without FME, but with a good workspace a piece of cake.
In addition to open data and project data, we also work a lot with analysis results. Then it concerns specific questions within the project. With FME, for example, we combine a KLIC dataset with the construction roads to be constructed. In this way you gain early insight into interfaces and possible bottlenecks in a project.
We have created various so-called FME workspaces for the above data types, each of which processes one or more data sets or performs an analysis on them. Such a workspace is a virtual scripting environment with various configured steps that follow each other (see also an image below). The result is then prepared for use in Clearly. We have also created a workspace for this. This workspace extracts the relevant parts for Clearly from all data and, if necessary, also adjusts the data immediately. In this way we can load data into Clearly quickly and reliably.
Looking for the right 'KLIC'
A good example of various datasets and questions that come together in a project is the use of the KLIC data. This dataset with cables and pipes (K&L) is obtained by submitting a KLIC report to the Land Registry. Recently, the land registry has made a considerable improvement that makes querying the data a lot easier. No more PDFs that need to be digitized to geometry, but geographical files (in GML format) that contain all geometry.
The (simple) workspace that we are now gradually building up reads all GML files from such a KLIC report and processes them into Clearly-specific data.
The image below shows part of that workspace.
Within the KLIC we find various types of K&L, in a number of cases with their own attributes. That is why we have set up a process for each type of K&L. In each process, geometry and attributes are selected and updated to a Clearly-ready dataset. Very specifically, the NullAttributeMapper transformer is also used. This transformer offers functionality that checks whether empty fields are actually empty and whether they are of a certain data type.
In practice, we often see that not all datasets are properly filled in and therefore cannot be read out. This once again makes it clear how important it is to have and keep data correct at the source.
Finally, the dataset, currently in Shapefile format, is checked by our Clearly workspace. Among other things, we look at:
the validity of the geometry;
the simplification of the geometry if desired;
a number of basic attributes for Clearly: are they present or do they still need to be created;
and we convert the dataset to the correct projection
The whole is then ultimately Clearly-proof.
…practice what you preach
In addition to this KLIC workspace, we have set up more of these types of processes. All these processes help to work effectively and efficiently in our Clearly projects. With FME, we also guarantee the quality and consistency of the data we supply.
In the future we see more possibilities for processing data with FME. Various services and services become possible with FME Server, for example. Internally, a number of manual processes can also be replaced by the use of FME.
All in all, Future Insight stands for a smart approach to (infrastructural) projects. We believe that we can make a significant contribution to this with Clearly. At the same time, we also do this internally with the commitment of FME: “…practice what you preach”. Working smarter and better is in our blood. Do you feel the click?
Want to know more?
Do you want to know more? Bas Hoorn is happy to tell you more about it.