According to the challenges of storage and computation faced by massive, multi-source and heterogeneous GNSS data, the design objective of cloud GNSS is analyzed, then the architecture of cloud GNSS from infrastructur...
详细信息
ISBN:
(纸本)9783662466315
According to the challenges of storage and computation faced by massive, multi-source and heterogeneous GNSS data, the design objective of cloud GNSS is analyzed, then the architecture of cloud GNSS from infrastructure, data management, service management to application is designed, the deployment model including service management platform, Web server cluster and multiple Hadoop clusters is provided, and its' characteristics such as strong expansibility, high reliability, loose coupling, are summarized. Cloud GNSS platform is built in the experiment, the storage model of massive GNSS data and the parallel computing model of GNSS network are built, distributed storage, parallel retrieval, sub-network division distributed computing and datapublication are achieved. The result shows that the architecture proposed by the paper can be applied in the storage, processing and service publication of large-scale GNSS network.
A distributedparallel database resource management method based on directed graph is proposed. By using content distributor based on distributed unstructured P2P network association, the high performance and stabilit...
A distributedparallel database resource management method based on directed graph is proposed. By using content distributor based on distributed unstructured P2P network association, the high performance and stability of distributedparallel database system in dynamic changing environment are guaranteed. The problem of network congestion caused by too many redundant messages and the 'barrel effect' caused by resource imbalance due to differential configuration is solved, through resource search algorithm based on directed graph lookahead, and the query node cached the resource information of two-level neighbor nodes. By adopting the Linux cgroups resource management mechanism, fully considering the multi tenant and multi factor based resource scheduling strategy, reduce resource fragments and better meet the problem of distributedparallel database storage or hot spot processing.
Public opinion information on the network is abundantly decentralized in Internet, these false ones, easily misleading, not legally binding, it is difficult to deal with. Quickly master information is a prerequisite t...
Public opinion information on the network is abundantly decentralized in Internet, these false ones, easily misleading, not legally binding, it is difficult to deal with. Quickly master information is a prerequisite to deal with the network of public opinion, clustering algorithm in data mining plays an important role in in the collection of information statistics. Disadvantages of classical clustering methods are high resources consumption, low efficiency, high time and space complexity, and difficult to deal with large-scale data processing in massive network. To the network public opinion text as the research object, in-depth study of summarizing the network public opinion monitoring technology based on distributed MapReduce, including parallel technology, improved clustering algorithm, relational database and distributed database construction, which is in order to improve the efficiency of information processing, to reduce the pin.
Digital signal processing as a key and difficult point of network technology research and development, currently commonly used content such as LabVIEW. But from a practical point of view, while these techniques can be...
Digital signal processing as a key and difficult point of network technology research and development, currently commonly used content such as LabVIEW. But from a practical point of view, while these techniques can be used to process real-time signals, they can't handle historical offline data. The Spark parallel computing studied in this paper can be used to process offline signals. Therefore, on the basis of understanding the development trend of Spark parallel computing framework, the distributed Mallat algorithm is analyzed based on Spark parallel computing engine, and the application performance of the corresponding algorithm is verified.
Understanding spatial and temporal dynamics of mobile internet services in cellular data network can be of great help to network management and service provisioning. To this end, we conduct the detailed measurement an...
详细信息
ISBN:
(纸本)9781629931357
Understanding spatial and temporal dynamics of mobile internet services in cellular data network can be of great help to network management and service provisioning. To this end, we conduct the detailed measurement analysis of spatial and temporal dynamics from point of view of traffic generated by subscribers. The analyzation is based on a large-scale data set collected from a commercial ISP covering an entire city in Southern China. We analyze individual subscriber behaviors and observe a significant variation in network usage among subscribers. We characterize mobile internet services spatial and temporal dynamics and identity their relation to traffic volume. The data set tracks more than 3 million mobile subscribers. To handle the problem of big data processing, the data are parallel processed using MapReduce programming model, a novel framework for distributed computing with proved high efficiency and low cost features. Generally, our observations deliver important insights into mobile internet service and mobile subscriber behavior.
On the premise of analyzing various evaluation methods and means and constructs the comprehensive evaluation model of external knowledge transfers risk of transformation enterprises. BP network has robustness and faul...
On the premise of analyzing various evaluation methods and means and constructs the comprehensive evaluation model of external knowledge transfers risk of transformation enterprises. BP network has robustness and fault tolerance, it is distributed storage of information, local damage will be a certain degree of reduce the network performance, does not result in a catastrophic risk, tolerance with a larger error in the input value, and even individual errors exist, at the same time each activity unit has the independence of the information processing, which can realize the parallel computing, improve speed.
The convolutional neural network training algorithm has been implemented for a central processing unit based high performance multisystem architecture machine. The multisystem or the multicomputer is a parallel machin...
The convolutional neural network training algorithm has been implemented for a central processing unit based high performance multisystem architecture machine. The multisystem or the multicomputer is a parallel machine model which is essentially an abstraction of distributed memory parallel machines. In actual practice, this model corresponds to high performance computing clusters. The proposed implementation of the convolutional neural network training algorithm is based on modeling the convolutional neural network as a computational pipeline. The various functions or tasks of the convolutional neural network pipeline have been mapped onto the multiple nodes of a central processing unit based high performance computing cluster for task parallelism. The pipeline implementation provides a first level performance gain through pipeline parallelism. Further performance gains are obtained by distributing the convolutional neural network training onto the different nodes of the compute cluster. The two gains are multiplicative. In this work, the authors have carried out a comparative evaluation of the computational performance and scalability of this pipeline implementation of the convolutional neural network training with a distributed neural network software program which is based on conventional multi-model training and makes use of a centralized server. The dataset considered for this work is the North Eastern University's hot rolled steel strip surface defects imaging dataset. In both the cases, the convolutional neural networks have been trained to classify the different defects on hot rolled steel strips on the basis of the input image. One hundred images corresponding to each class of defects have been used for the training in order to keep the training times manageable. The hyperparameters of both the convolutional neural networks were kept identical and the programs were run on the same computational cluster to enable fair comparison. Both the convolutional neur
At present, the short-term power load prediction model generally uses the traditional wavelet neural network for prediction, which utilizes the gradient descent algorithm. However, it has the problems of sensitivity t...
At present, the short-term power load prediction model generally uses the traditional wavelet neural network for prediction, which utilizes the gradient descent algorithm. However, it has the problems of sensitivity to the initial value and low prediction accuracy. To address this issue, we build a novel short-term power load prediction model leveraging the Wavelet Neural network (WNN) base on the Comprehensive Improved Shuffled Frog Leaping Algorithm (CSFLA). By using this prediction model, we firstly conduct distributed storage and processing of a large amount of preprocessed historical load data, and then parallelize the processed historical load data by using MapReduce programming framework and WNN to obtain the prediction results. In the experiments, simulation results demonstrate that the proposed prediction model has high accuracy, strong adaptability and excellent parallel performance.
OpenFlow switches in SDN use Multiple Flow Tables (MFTs) for fine-grained flow control. Commodity switches integrate hardware storage resources such as SRAM and TCAM to store flow tables to achieve high-speed lookups....
OpenFlow switches in SDN use Multiple Flow Tables (MFTs) for fine-grained flow control. Commodity switches integrate hardware storage resources such as SRAM and TCAM to store flow tables to achieve high-speed lookups. Many increased flow tables are rapidly exhausting these hardware storage resources, which makes the switches have to balance high-speed search and massive storage. The rule-caching scheme is a popular method to solve this problem, which caches the most commonly used rules into hardware storage resources. The existing rule-caching schemes are based on single hardware storage resources, and they cannot flexibly adjust the caching strategy according to the traffic characteristics. Simultaneously, the deployed commodity switches face the problem of difficulty in changing the size of SRAM and TCAM. This paper innovatively proposes the MixedCache scheme, which makes full use of the hardware storage resources in the switch according to the skewed characteristics of network traffic. MixedCache stores the large flows in SRAM by exact match and stores the small flows in the TCAM by wildcard match. MixedCache does not need to change the size of the deployed switch hardware storage resources, but makes full use of existing resources. Compared with the rule-caching scheme based on the exact match, the cache hit rate can increase by up to 15.61%. Compared with the rule-caching scheme based on the wildcard match, the cache hit rate can increase by up to 29.69%.
The first level trigger of LHCb accepts one million events per second. After preprocessing in custom FPGA-based boards these events are distributed to a large farm of PC-servers using a high-speed Gigabit Ethernet net...
The first level trigger of LHCb accepts one million events per second. After preprocessing in custom FPGA-based boards these events are distributed to a large farm of PC-servers using a high-speed Gigabit Ethernet network. Synchronisation and event management is achieved by the Timing and Trigger system of LHCb. Due to the complex nature of the selection of B-events, which are the main interest of LHCb, a full event-readout is required. Event processing on the servers is parallelised on an event basis. The reduction factor is typically 1/500. The remaining events are forwarded to a formatting layer, where the raw data files are formed and temporarily stored. A small part of the events is also forwarded to a dedicated farm for calibration and monitoring. The files are subsequently shipped to the CERN Tier0 facility for permanent storage and from there to the various Tier1 sites for reconstruction. In parallel files are used by various monitoring and calibration processes running within the LHCb Online system. The entire data-flow is controlled and configured by means of a SCADA system and several databases. After an overview of the LHCb data acquisition and its design principles this paper will emphasize the LHCb event filter system, which is now implemented using the final hardware and will be ready for data-taking for the LHC startup. Control, configuration and security aspects will also be discussed.
暂无评论