Data Integration – there is a better way

We have talked about the importance of data and data integration in many of our recent blogs, however we have never really discussed how we do it, so we thought it was about time we did.

Over the last few years, we have been working on a project for one of our mining services clients which involves integrating data from a number of different sources (Excel spreadsheets, Cloud and On-premise applications, Machine/Vehicle Sensor Data etc.) into a centralised data warehouse which is then used for invoicing, budgeting, forecasting, reporting and analytics.

On this project we worked together with another local company, who specialise in data science, AI and machine learning. We have recently formed a closer relationship with this company which enables us, together, to provide our clients with a full End to End Business Operations System (EEBOS). At the core of the EEBOS solution are two distinct tools, Datahub and Mailbot.  These technologies specifically addresses the data integration we are talking about.  This integrated data is then used to drive the reporting, analytics and budgeting/forecasting ‘front- end’ which the user sees and interacts with.

But what exactly is Datahub and Mailbot?

Datahub

DataHub was built from the ground up to seamlessly synchronise data from the various sources and systems. Datahub’s core engine automates record matching and data translations using state-of-the-art algorithms. Data then smoothly flows between synchronised systems to ensure all updates are translated and sent where they need to be. Administrators are kept in the loop with scheduled reports, notifications of edit conflicts via email and the ability to fine-tune the automatically learned translation mappings.

DataHub’s system-specific connectors means adding new systems to synchronise is not a problem cloud-based applications, on-premises systems and real-time (sensor) feeds are all able to be synchronised. Swapping systems from one vendor to another also becomes much smoother since DataHub seamlessly takes care of all the translations and matching, allowing clients to choose best-of-breed systems for each  function of your business and avoid vendor lock-in.

Also, when you need to put together data quickly to keep operations moving, you need to use whatever is at hand – whether it’s Excel spreadsheets, reports or CSV file dumps, you work with what can get you there. But afterwards you’ve got a problem: all these different pieces need to be consolidated back together so that you can summarise and quantify what has actually been happening and determine what needs to be done next.

Need to check the raw data that’s in the data warehouse? DataHub has a fully secure web site that lets authorised users view any data in the system. You can even have it configured so that administrative staff can manage the data directly in the DataHub, letting you store and maintain supporting data that you need but isn’t in your Excel spreadsheets.

DataHub isn’t just a data warehouse, it’s also a complete data integration and synchronisation suite that lets you share data across all your business systems, from third-party software packages to cloud-based apps, corporate data warehouses and even external partner’s IT systems. Make double entry a thing of the past and bring together all your data into a cohesive unit for whole-of-business reporting. It even makes swapping out one vendor’s system for another a seamless experience, so you can choose best-of-breed products for your company.

Mailbot

Most IT solutions expect you to manually convert your data into a format and naming convention that they will accept before allowing bulk import, or worse force you to manually type the data into their product. Wouldn’t it be great if you could instead just email your spreadsheets and reports to a central repository and dashboards and summary reporting? All by nothing more than sending an email with an attachment? Some of you would have heard of the term ’email scraping’.  In essence this is exactly what the MailBot is designed to do.

MailBot is an autonomous data-crunching robot (installed as a Windows service in your IT network) that monitors one or more mailboxes for incoming data in emails (Microsoft Exchange / Office 365 is natively supported). MailBot reads any attachments and automatically processes them, converting and validating the attachment’s contents into structured data and saving it to the DataHub data warehouse. Any dashboards or reports that run off of the data warehouse will immediately pick up the new data – all triggered from a simple email and without a single change to your existing processes.

So how does it work? We know that every company’s data will reflect their unique way of operating, so a one-size-fits-all approach to data processing will never work. MailBot is designed to handle this, being easily extensible with custom connectors tailored specifically to your data formats. Hence there is no need to re-format your spreadsheets.

MailBot will detect errors in data, collate them and automatically send error reports via email to the people who need to know. Fixing the problems and re-emailing in the attachment is all that is necessary to update the DataHub warehouse.

If you make a mistake, just fix it in your spreadsheet and re-email it in – MailBot will automatically notice what is an edit of existing data versus new records and update the warehouse accordingly.

With Datahub and Mailbot you can streamline your process without having to change your process.

To find out more about how Datahub and Mailbot can help your business get in contact with us.