Practical use of Cognitive Computing

If you combine artificial intelligence with signal processing, you can improve everyday tasks such as safety equipment checks, anomaly detection, or reading documents. Technology incorporating machine learning, natural language processing, human-computer interaction, and more are described as cognitive computing. At the first sight, it may seem like too much science behind, yet the algorithms can be implemented easier than you might think and help you with your marketing or occupational safety within your business.

We started the practical part of the workshop with an overview of the areas of Azure Cognitive Services and then we moved to use cases of Anomaly Detector, text, and vision services.

Azure Cognitive Services

Before we explain the practical use of Azure Cognitive Services, introducing all areas can give a good picture of how technology collaboration and accessibility can create an empowering experience for the end-user.

Azure Cognitive Services is a set of machine learning algorithms developed to solve problems in the field of Artificial Intelligence (AI). They are available widely to developers without requiring machine-learning expertise. All it takes is an API call to embed the ability to see, hear, speak, search, understand, and accelerate decision-making. It is a package of 25 tools that through APIs allow the developers to add a variety of features to their applications.

Part 1| Anomaly Detector

In the first part Azure Cognitive Services demos in practice, we focused on the Anomaly Detector service and AML Notebooks.

In general, the anomaly detection algorithm predicts the value at a selected point based on previous observations. This prediction always has a certain sensitivity/confidence interval in which it moves (light blue area in the image below).

The predicted value is then compared to the actual measured value and the points at which it is actually measured are identified.

The Azure anomaly detector REST API is a service that provides the ability to detect anomalies in time series from any data source. The only condition is that the data must be in a defined structure: timestamp + selected value

The Rest API can be called from any tool that can call GET requests. Examples include Jupyter Notebooks, Postman, Visual Studio, or Azure-enabled services such as Databricks, or Azure Machine Learning with integrated notebooks.

Azure anomaly detection REST API is a service provided by Azure as one of the Cognitive services accessible via the Azure portal where you can obtain the authorization key and endpoint that are necessary for later calls. The output after calling this API is a set of parameters that say whether it is an anomaly at a given point in time and whether it was a decrease or an increase.

Azure anomaly detector REST API works in two optional modes.

"Last" mode Works on stream data. Each point in time is analyzed based on a model trained by previous data. For example, it is suitable for data from Google Analytics.

"Entire" mode creates and trains one model for the entire time series and detects anomalies over the entire model and all data points at once.

Anomaly detection can be configured by entering parameter values in the request. We have the ability to work with these parameters:

• Sensitivity - "Sensitivity" is from 0 to 99, defines how sensitively the back-end API detects anomalies
• Granularity - It can be annual, monthly, weekly, daily, hourly, minute (data must correspond to this granularity, i.e. the difference between subsequent data points must correspond to this Period)
• CustomInterval - For cases of different granularity (e.g. customInterval = 5 & granularity = minute will have an interval of 5 minutes)
• Period - defines how many history points are used to detect current anomalies. The extent of the period will vary according to the granularity.
• maxAnomalyRatio - "MaxAnomaly" defines the maximum percentage of anomalies in one detection.

We also talked about the use of the Anomaly detector REST API in the event of an error in the implementation of measurement on the web.

Business Case – Fooled bidding engine

Let’s look at one of our client’s normal day, an e-shop that sells clothes and uses a bidding engine to place ads. If everything works by default, the customer sees the advertisement, comes to the website, and makes an order.

Information about a successful order is sent to the bidding engine, which optimizes its behavior based on it. Unfortunately, not all days are so exemplary.

Unfortunately, a new version of the site was published from the test environment, which contained a measurement error that caused a duplication of pageviews to be measured for the men's sweatshirt category, indicating a successful conversion. This caused the bidding engine to start receiving false information and to make bad decisions because it thought the men's sweatshirt ad is twice as successful as it actually is. The engine started investing higher amounts in the ad. If this error would not be caught in time, it could lead to large losses in the marketing budget.

In order to detect an anomaly, it is necessary to have a specialist who would check the data regularly and would be sufficiently attentive to changes in the data. It may not be as easy as noticing the mistakes in this case (in Google Analytics, only a week is visible by default).

Solution - Detecting Anomalies

Such a fluctuation in the measured data can be detected with Azure Anomaly Detector. There are two options.

Azure ML

One option is to use the Azure Machine Learning tool using integrated notebooks. Among other things, this tool allows you to create and manage models for machine learning.

We integrated this tool into the standard e-commerce process. Data for anomaly detection were obtained from the web analytics system in the standardized format Timestamp, Value. We analyzed the data by calling the Azure anomaly detector API, and if we detect a problem, we start fixing it immediately.

However, this particular solution comparing to similar tools can be challenging due to:

• Need for regular manual execution - Someone needs to run the code regularly
• The necessity to have relatively advanced knowledge of some programming language such as Python
• Not completely clear insights, it is necessary to further modify, visualize, etc.

Waaila

The second option, which eliminates the problems described above, is to use the Waaila app.

Waaila's connection to the standard data processing system is similar to the previous solution. We can connect directly to the data in GA through it. It is then possible to perform various tests on these data using logical conditions. In our case, we defined data in the format Timestamp: pageviews. Then the Azure anomaly detector API is called, and we find out if there is an anomaly in the dataset.

Why is the Waaila app different

The result obtained through Waaila has a clear and simple form. In the output table, we can see the days on which the anomaly was detected. These are the days that followed the publication of a new version of the website containing a measurement error. We can see that on both days there was a positive anomaly, which means that the display of the confirmation page was more frequent than was predicted based on previous developments.

Part 2 | Azure Cognitive Services – Text & Vision

The second part of the workshop contained examples of the Text and Vision group of Cognitive services. One of the most common use cases involves the Vision group of Cognitive Services. We meet frequently with companies who experience troubles with employee safety obedience and are looking for a more sophisticated solution. The second case focuses on text recognition, for instance, homework check automation or reading receipts. Third case comments on means to gather data about customer satisfaction.

Business Case 1: Safety Equipment Check

Construction companies are required to check that all workers on their sites wear hard hats and reflective vests. However, this is costly in terms of time and human resources. To minimize this cost, companies can incorporate an AI solution using the Custom Vision service in order to detect the equipment automatically.

Using this approach, when somebody enters the construction site, the security camera at the entrance sends their image to be processed using the Custom Vision service. The image is evaluated for the presence of safety equipment based on a pre-trained model of similar tagged images. The results of the evaluation are sent back and in case of missing equipment, a message is outputted.

The visual evaluation can be combined with customized and personalized messages to be more noticeable and thus a better warning for workers without the safety equipment. The customized messages can be created in real-time using the Text to Speech service. This service constructs a voice message from inputted text using a high variety of voices in over 45 languages (including the Czech language).

Moreover, to personalize the warnings, Face verification can be used to compare the image of entering workers with database workers' photos. Based on the extracted name, a personalized message can be constructed again using the Text to Speech service. The personalized messages have the highest impact on ensuring that workers wear the safety equipment, especially if combined with the possibility to report the outcome to the supervisor.

This AI solution to automatic check allows the companies to save human and financial resources while minimizing the risk of having to pay a fine for potentially incorrect safety equipment on the site.

Business case 2: Automatic evaluation of exercise results

AI is very useful in areas where documents were not fully converted to electronic form yet. When the current Covid-19 pandemic closed schools, teachers were often limited by the fact that their teaching materials were not well equipped for distance learning. Correcting homework in exercise books distantly requires a series of printing and scanning, costing a high amount of time and other resources. This opens a possibility for the AI solution using a Form Recognizer.

Form Recognizer allows you to automatically extract both printed and hand-written text from computer non-readable documents and images. You provide training files and select one of the available approaches. The first approach consists of optimized prepared models for receipts and business cards, however, these cannot be applied to other documents. The second approach allows extracting all text fields that can be viewed as key-value pairs. While this provides more flexibility than the first approach, you cannot select which part of the file to concentrate on and cannot extract values without a well-located key. Most flexible and thus most suited for the case of school material is the third approach which consists of labeling required text fields and training on thus labeled files. For labeling, there is a special online tool where you can create labels and interactively assign them to OCR-extracted text fields from your training files. For illustration, we labeled and trained a simple model on a page from a first-grade mathematics textbook. Below is a snapshot of information extracted based on the simple model, displayed both visually in colored rectangles and as a list of values assigned to the tags of matching color along with a confidence of the assignment.

Using the Form Recognizer you can construct automatic homework evaluation by collecting a sample of homework, labeling the required fields on the sample, and training the model. Then you can use this model to form automatic extraction of the required information from other files and comparison of the extracted values to a solution key. While this is too demanding for a single teacher, it opens a business opportunity to cooperate with a publishing house on providing an official solution thus helping distance teaching.

Applications of Form Recognizer

Form Recognizer is useful in many fields. For example, when a customer brings a receipt for purchased goods he wants to complain about, the cashier needs to retype the receipt to fill in the complaint. Alternatively, in a loyalty program when the producer company requires invoices from distributors as proof for distributing the products, they often receive the forms in a scan or even paper form that they need to digitalize. To sum up, Form Recognizer saves both time and other resources. Based on our experience, it can provide even better results than manual extraction as the people working on it may often be over-worked or not well informed.

Business case 3: Customer satisfaction

Information about customer satisfaction is necessary for improving the quality of goods and services and keeping customers from going to the competition. It is mostly gathered from surveys and a set of buttons with smiley faces, however, these provide not only under-represented but also skewed results due to the selection of people willing to answer it. To overcome this problem, companies can employ behavior analytics on the e-shop and perceived emotion recognition in the stores. The e-shop Text Analytics service can help process comments and chat messages to prevent customers from leaving due to negative experiences or negative impressions of one customer to spread to other customers. At this moment there is only a selected number of language options for the TextAnalytics but it can be paired with Translator service to cover other languages. The text extraction can be combined with an analysis of customers' paths, waiting times, and other behavioral patterns which may provide further information on satisfaction.

In stores, instead of using the buttons to express satisfaction, companies can use the entrance cameras to take an image. The image is then processed using a part of the Face service that can evaluate the Perceived emotion recognition. In this recognition, the location of the face in the image is found and several emotions are searched for in the detected face to evaluate the degree to which they are recognizably present on the face. This can be then used to find out how happy was a customer when leaving or how much did his mood change while inside the store which can help with the optimization of the store and the services.

Summary

To summarized the workshop focused on Cognitive Computing and its practical applications, we showed and explain a few use interesting cases for everyday use. With Azure, you can not only ensure security but also it can be very easily connected to cloud storage and other tools that overall create one well-working and inter-connected environment, built for your convenience.

We have applied similar and many more solutions. Marketing combined with Data Science experience and technical expertise provides us with a competitive advantage to build custom solutions for each project.

Let us know, how we can help you grow.