Big Data

Copy/Replicate of Big Data over Internet

The project leader is Andrey Y Shevel

The Big Data is the technical term which denotes the large volume of data. The volume is so large that it is not easy to copy or transfer such the volume.

It is clear that the size of Big Data depends on the time when you are talking on the topic. For example, in 2013 we can say about big data after size around hundreds of TeraBytes (TB).

What are sources of Big Data? Many areas of human activity. We can remind experimantal data from High Energy Physics and Astrophysics, Genetics, Biology, etc. For example, you can take a look at the page [Information Revolution: Big Data Has Arrived at an Almost Unimaginable Scale]. 

For now we are planning to undertake comparative study of existing systems to copy/replicate Big Data over Internet and to develop the set of criteria to compare different systems of copying Big Data.

Later on we plan to suggest architecture of copy/repicate system for Big Data for a range of application areas: communications, scientific physics experiments, etc. 

A range of related topics

• Big data at NIST
  
• Летняя суперкомпьютерная академия
• Status Report of the DPHEP Study Group: Towards a Global Effort for Sustainable Data Preservation in High Energy Physics - Arxiv (2012)
  

• LHC plans for open data future - Nature Nov 2013

• Study Group on Data Preservation and Long Term Analysis in High Energy Physics

• Data and Software Preservation for Open Science
• PerfSonar - www.perfsonar.net

• Welcome to the Linux Kernel MultiPath TCP project

• Experiences with MPTCP in an intercontinental multipathed OpenFlow network

Welcome to the eduPERT Knowledge Base!!!!

Some references for Big Data transfer

• Lavanya Ramakrishnan, Chin Guok, Keith Jackson, Ezra Kissel, D. Martin Swany, Deborah Agarwal // On-demand Overlay Networks for Large Scientific Data Transfers // 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing // Melbourne, VIC, Australia // May 17-May 20 // ISBN: 978-0-7695-4039-9 // http://www.computer.org/csdl/proceedings/ccgrid/2010/4039/00/4039a359-abs.html

• Large scale scientific data transfers are central to scientific processes. Data from large experimental facilities have to be moved to local institutions for analysis or often data needs to be moved between local clusters and large supercomputing centers. In this paper, we propose and evaluate a network overlay architecture to enable high-throughput, on-demand, coordinated data transfers over wide-area networks. Our work leverages Phoebus and On-demand Secure Circuits and Advance Reservation System (OSCARS) to provide high performance wide-area network connections. OSCARS enables dynamic provisioning of network paths with guaranteed bandwidth and Phoebus enables the coordination and effective utilization of the OSCARS network paths. Our evaluation shows that this approach leads to improved end-to-end data transfer throughput with minimal overheads. The achieved throughput using our overlay was limited only by the ability of the end hosts to sink the data.

• How to transfer large amounts of data via network (by Harry Mangalam
  <harry.mangalam@uci.edu>)

• Big Data Preservation - presentation "Big Data and Long-Term Preservation: Challenges and Opportunities" by Jamie Shiers
  
• Enabling high throughput in widely distributed data
  management and analysis systems: Lessons from the LHC

• High Performance SSH/SCP - HPN-SSH

• Fast Data Transfer - FDT

• Guide to Bulk Data Transfer over a WAN (by Brian Tierney and Joe Metzger, ESnet)

• BBFTP - utility for bulk data transfer. bbFTP is a file transfer software. It implements its own transfer protocol, which is optimized for large files (larger than 2GB) and secure as it does not read the password in a file and encrypts the connection information. bbFTP main features are:

• Encoded username and password at connection

• SSH and Certificate authentication modules

• Multi-stream transfer

• Big windows as defined in RFC1323

• On-the-fly data compression

• Automatic retry

• Customizable time-outs

• Transfer simulation

• AFS authentication integration

• RFIO interface

• BBCP – utility to trasfer the data over network (Andrew Hanushevsky abh@stanford.edu )

• ESnet Fasterdata Knowledge Base

• PHEDEX -

• Berkeley Storage Manager (BeStMan)

• FTS3 - file transfer service
  
  

• GridFTP - Grid/Globus data transfer tool. Client part is known as globus-url-copy.

• Efficient Data Transfer Protocols for Big Data - Brian Tierney, Ezra Kissel, Martin Swany, Eric Pouyoul

  • Lawrence Berkeley National Laboratory, Berkeley, CA 94270

  • School of Informatics and Computing, Indiana University, Bloomington, IN 47405

  • Abstract—Data set sizes are growing exponentially, so it is important to use data movement protocols that are the most efficient available. Most data movement tools today rely on TCP over sockets, which limits flows to around 20Gbps on today’s hardware. RDMA over Converged Ethernet (RoCE) is a promising new technology for high-performance network data movement with minimal CPU impact over circuit-based infrastructures. We compare the performance of TCP, UDP, UDT, and RoCE over high latency 10Gbps and 40Gbps network paths, and show that RoCE-based data transfers can fill a 40Gbps path using much less CPU than other protocols. We also show that the Linux zero-copy system calls can improve TCP performance considerably, especially on current Intel “Sandy Bridge”-based PCI Express 3.0 (Gen3) hosts.

• Big Data @ NIST.GOV
• Big Data @ San Diego Supercomputer Center
  
• Big Data @ HP
• Big Data @ IBM
• Big Data @ Oracle
  
• Picture of Big Data from IBM point of view

Conferences

•  https://tnc2013.terena.org