Modern Big Data Architectures

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla facilisi. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Agent mining is an interdisciplinary field, which combines efforts of multi‐agent systems, data mining, machine learning, and other related fields. The motivation to join these fields is to take advantage of known ML/DM algorithms on the one hand, while utilizing flexibility and efficiency in distributed computation provided by multi‐agent systems. Cao et al..

Zhang et al. list the following advantages of using agents for DDM tasks:

retaining the autonomy of the data sources

facilitating interactive distributed data mining

improving dynamic selection of sources and data gathering

having high scalability to massive distributed data

stimulating multi‐strategy distributed data mining

enabling collaborative data mining

In Liu et al. the authors propose a novel DDM model called DRHPDM (Data source Relevance‐based Hierarchical Parallel Distributed data mining Model).

The model utilizes Web Service technology for seamless integration of components. Depending on the data relevance the sources are aggregated into Local Centralized Data Mining Layer (LCDML) or Local Parallel Data Mining Layer (LPDML). A Global Processing Unit (GPU) integrates the results.

The DRHPDM mining workflow consists of the following steps:

Submission of DM requirements to the GPU by the user

The DM request is divided into a number of DM sub‐tasks

Distribution of subtasks to the sites

The site receives tasks and registers data sets at GPU

Data set information is combined and grouped logically

Sites are notified to start DM processes

The DM results (models) are transferred to Local Managing Agent (LMA) for local integration

The final global model is transformed and submitted to the user

Another example of agent application in DDM is presented in Chaimontree et al., where a Multi‐Agent Based Clustering (MABC) framework is described. The clustering process consists of two parts. Firstly, based on the specific clustering algorithms, the initial clustering configuration is generated. In the second phase agents negotiate with each other to improve the initial configuration.

Four types of agent are distinguished in MABC:

user agents

data agents

validation agents

clustering agents

User agents provide the interface to the system. They receive the request, spawn clustering agents and return the result. Data agents manage the available data sources. There is one data agent dedicated for each source. Clustering agents run the main clustering algorithms. Finally, validation agents validate and return results.