2022
21.
Shridhar, Ayush; Nadig, Deepak
Heuristic-based Resource Allocation for Cloud-native Machine Learning Workloads Proceedings Article
In: 2022 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), pp. 415-418, 2022, ISSN: 2153-1684.
Abstract | BibTeX | Tags: | Links:
@inproceedings{10227727,
title = {Heuristic-based Resource Allocation for Cloud-native Machine Learning Workloads},
author = {Ayush Shridhar and Deepak Nadig},
url = {https://deepaknadig.com/wp-content/uploads/2023/10/Heuristic-based_Resource_Allocation_for_Cloud-native_Machine_Learning_Workloads.pdf},
doi = {10.1109/ANTS56424.2022.10227727},
issn = {2153-1684},
year = {2022},
date = {2022-12-01},
urldate = {2022-12-01},
booktitle = {2022 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS)},
pages = {415-418},
abstract = {As machine learning workloads become computationally demanding, there is an increased focus on distributed machine learning to train and deploy models across multiple machines in a cloud-native cluster. However, optimizing a machine learning model’s lifecycle to facilitate efficient resource utilization is still an active area of research. The approach typically involves a manual effort to partition the models into distinct layers and decide how to store these distinct layers on a distributed computing framework. However, distributing distinct layers across nodes can induce a network latency bottleneck in the machine learning pipeline. Further, the above process becomes more inefficient as models become increasingly complex. In this paper, we present a heuristic-based approach to distributed model training. Further, we analyze the resource utilization metrics from a sample machine learning pipeline deployed on a KubeFlow MLOps framework testbed.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
As machine learning workloads become computationally demanding, there is an increased focus on distributed machine learning to train and deploy models across multiple machines in a cloud-native cluster. However, optimizing a machine learning model’s lifecycle to facilitate efficient resource utilization is still an active area of research. The approach typically involves a manual effort to partition the models into distinct layers and decide how to store these distinct layers on a distributed computing framework. However, distributing distinct layers across nodes can induce a network latency bottleneck in the machine learning pipeline. Further, the above process becomes more inefficient as models become increasingly complex. In this paper, we present a heuristic-based approach to distributed model training. Further, we analyze the resource utilization metrics from a sample machine learning pipeline deployed on a KubeFlow MLOps framework testbed.
20.
Nadig, Deepak; Ramamurthy, Byrav; Bockelman, Brian
SNAG: SDN-Managed Network Architecture for GridFTP Transfers Using Application-Awareness Journal Article
In: IEEE/ACM Transactions on Networking, vol. 30, no. 4, pp. 1585-1598, 2022, ISSN: 1558-2566.
Abstract | BibTeX | Tags: | Links:
@article{9718030,
title = {SNAG: SDN-Managed Network Architecture for GridFTP Transfers Using Application-Awareness},
author = {Deepak Nadig and Byrav Ramamurthy and Brian Bockelman},
doi = {10.1109/TNET.2022.3150000},
issn = {1558-2566},
year = {2022},
date = {2022-08-01},
journal = {IEEE/ACM Transactions on Networking},
volume = {30},
number = {4},
pages = {1585-1598},
abstract = {Increasingly, academic campus networks support large-scale data transfer workflows for data-intensive science. These data transfers rely on high-performance, scalable, and reliable protocols for moving large amounts of data over a high-bandwidth, high-latency network. GridFTP is a widely used protocol for wide area network (WAN) data movement. However, as the GridFTP protocol does not share connection information with the network-layer, network operators have reduced flexibility, particularly in identifying/managing flows across the network. We address this problem by deploying a production “ application-aware ” software defined network (SDN) for managing GridFTP transfers for data-intensive science workflows. We first propose a novel application-aware architecture called SNAG (SDN-managed Network Architecture for GridFTP transfers). SNAG combines application-layer and network-layer collaboration (termed “application-awareness”) with SDN-enabled network management to classify, monitor and to manage network resources actively. Until now, our SNAG deployment has successfully classified over 1.5 Billion GridFTP connections at the Holland Computing Center (HCC), University of Nebraska-Lincoln (UNL). Next, we develop an application-aware SDN system to provide differentiated network services for distributed computing workflows. At HCC, we also demonstrate how our system ensures the quality of service (QoS) for high-throughput workflows such as Compact Muon Solenoid (CMS) and Laser Interferometer Gravitational-Wave Observatory (LIGO). Further, we also demonstrate how application-aware SDN can be exploited to create policy-driven approaches to achieve accurate resource accounting for each workflow. We present strategies for implementing differentiated network services and discuss their capacity improvement benefits. Lastly, we provide some guidelines and recommendations for developing application-aware SDN architectures for general-purpose applications.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Increasingly, academic campus networks support large-scale data transfer workflows for data-intensive science. These data transfers rely on high-performance, scalable, and reliable protocols for moving large amounts of data over a high-bandwidth, high-latency network. GridFTP is a widely used protocol for wide area network (WAN) data movement. However, as the GridFTP protocol does not share connection information with the network-layer, network operators have reduced flexibility, particularly in identifying/managing flows across the network. We address this problem by deploying a production “ application-aware ” software defined network (SDN) for managing GridFTP transfers for data-intensive science workflows. We first propose a novel application-aware architecture called SNAG (SDN-managed Network Architecture for GridFTP transfers). SNAG combines application-layer and network-layer collaboration (termed “application-awareness”) with SDN-enabled network management to classify, monitor and to manage network resources actively. Until now, our SNAG deployment has successfully classified over 1.5 Billion GridFTP connections at the Holland Computing Center (HCC), University of Nebraska-Lincoln (UNL). Next, we develop an application-aware SDN system to provide differentiated network services for distributed computing workflows. At HCC, we also demonstrate how our system ensures the quality of service (QoS) for high-throughput workflows such as Compact Muon Solenoid (CMS) and Laser Interferometer Gravitational-Wave Observatory (LIGO). Further, we also demonstrate how application-aware SDN can be exploited to create policy-driven approaches to achieve accurate resource accounting for each workflow. We present strategies for implementing differentiated network services and discuss their capacity improvement benefits. Lastly, we provide some guidelines and recommendations for developing application-aware SDN architectures for general-purpose applications.
19.
Lunar, Mohammad M. R.; Sun, Jianxin; Wensowitch, John; Fay, Michael; Tulay, Halit Bugra; Karanam, Venkat Sai Suman Lamba; Qiu, Brian; Nadig, Deepak; Attebury, Garhan; Yu, Hongfeng; Camp, Joseph; Koksal, Can Emre; Pompili, Dario; Ramamurthy, Byrav; Hashemi, Morteza; Ekici, Eylem; Vuran, Mehmet C.
OneLNK: One Link to Rule Them All: Web-Based Wireless Experimentation for Multi-Vendor Remotely Accessible Indoor/Outdoor Testbeds Proceedings Article
In: Proceedings of the 15th ACM Workshop on Wireless Network Testbeds, Experimental Evaluation & CHaracterization, pp. 85–92, Association for Computing Machinery, New Orleans, LA, USA, 2022, ISBN: 9781450387033.
Abstract | BibTeX | Tags: IRIS, Remote Wireless, Testbed, USRP, VM, Web Technologies | Links:
@inproceedings{10.1145/3477086.3480835,
title = {OneLNK: One Link to Rule Them All: Web-Based Wireless Experimentation for Multi-Vendor Remotely Accessible Indoor/Outdoor Testbeds},
author = {Mohammad M. R. Lunar and Jianxin Sun and John Wensowitch and Michael Fay and Halit Bugra Tulay and Venkat Sai Suman Lamba Karanam and Brian Qiu and Deepak Nadig and Garhan Attebury and Hongfeng Yu and Joseph Camp and Can Emre Koksal and Dario Pompili and Byrav Ramamurthy and Morteza Hashemi and Eylem Ekici and Mehmet C. Vuran},
url = {https://doi.org/10.1145/3477086.3480835},
doi = {10.1145/3477086.3480835},
isbn = {9781450387033},
year = {2022},
date = {2022-01-01},
booktitle = {Proceedings of the 15th ACM Workshop on Wireless Network Testbeds, Experimental Evaluation & CHaracterization},
pages = {85–92},
publisher = {Association for Computing Machinery},
address = {New Orleans, LA, USA},
series = {WiNTECH'21},
abstract = {As evolving wireless network architectures become more diverse, complex, and interdependent, and equipment costs prohibit broad access to such networks, remotely accessible experimental testbeds are gaining interest in recent years in wireless communication and networking research. This interest has exacerbated in 2020 and became a vital need during the current global pandemic. However, providing end-users of various educational backgrounds access to radio devices from a heterogeneous set of vendors is challenging. This paper introduces OneLNK, a remotely accessible testbed consisting of radio devices from three different vendors and developed using open source cloud-native technologies. End-users can access the functionalities of OneLNK from a single webpage without any local installations. Using the web URL, users can operate radio devices, set experiment parameters, observe results in real-time, and save generated experiment data for all radio devices. The interactive web UI and its working mechanism for supporting radio equipment are covered with specific experiment capabilities. A diverse set of radio equipment (mmWave, sub-GHz SDR, and sub-6GHz SDR) are facilitated to explain these capabilities. Moreover, measurements of path loss, Received Signal Strength (RSS), and Signal-to-Noise Ratio (SNR) using devices from three different vendors operating on a vast spectrum (568 MHz, 5.8 GHz, and 60 GHz) are reported. The majority of the remotely accessible OneLNK platform was developed remotely during the pandemic by a team of experts from five U.S. states.},
keywords = {IRIS, Remote Wireless, Testbed, USRP, VM, Web Technologies},
pubstate = {published},
tppubtype = {inproceedings}
}
As evolving wireless network architectures become more diverse, complex, and interdependent, and equipment costs prohibit broad access to such networks, remotely accessible experimental testbeds are gaining interest in recent years in wireless communication and networking research. This interest has exacerbated in 2020 and became a vital need during the current global pandemic. However, providing end-users of various educational backgrounds access to radio devices from a heterogeneous set of vendors is challenging. This paper introduces OneLNK, a remotely accessible testbed consisting of radio devices from three different vendors and developed using open source cloud-native technologies. End-users can access the functionalities of OneLNK from a single webpage without any local installations. Using the web URL, users can operate radio devices, set experiment parameters, observe results in real-time, and save generated experiment data for all radio devices. The interactive web UI and its working mechanism for supporting radio equipment are covered with specific experiment capabilities. A diverse set of radio equipment (mmWave, sub-GHz SDR, and sub-6GHz SDR) are facilitated to explain these capabilities. Moreover, measurements of path loss, Received Signal Strength (RSS), and Signal-to-Noise Ratio (SNR) using devices from three different vendors operating on a vast spectrum (568 MHz, 5.8 GHz, and 60 GHz) are reported. The majority of the remotely accessible OneLNK platform was developed remotely during the pandemic by a team of experts from five U.S. states.
2021
18.
Hu, Boyang; Nadig, Deepak; Ramamurthy, Byrav
Improving Service Performance through Multilayer Routing and Service Intelligence in a Network Service Mesh Proceedings Article
In: 2021 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), pp. 420-425, 2021, ISSN: 2153-1684.
Abstract | BibTeX | Tags: | Links:
@inproceedings{9936995,
title = {Improving Service Performance through Multilayer Routing and Service Intelligence in a Network Service Mesh},
author = {Boyang Hu and Deepak Nadig and Byrav Ramamurthy},
doi = {10.1109/ANTS52808.2021.9936995},
issn = {2153-1684},
year = {2021},
date = {2021-12-01},
booktitle = {2021 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS)},
pages = {420-425},
abstract = {Network service mesh architectures, by interconnecting cloud clusters, provide access to services across distributed infrastructures. Typically, services are replicated across clusters to ensure resilience. However, end-to-end service performance varies mainly depending on the service loads experienced by individual clusters. Therefore, a key challenge is to optimize end-to-end service performance by routing service requests to clusters with the least service processing/response times. We present a two-phase approach that combines an optimized multi-layer optical routing system with service mesh performance costs to improve end-to-end service performance. Our experimental strategy shows that leveraging a multi-layer architecture in combination with service performance information improves end-to-end performance. We evaluate our approach by testing our strategy on a service mesh layer overlay on a modified continental united states (CONUS) network topology.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Network service mesh architectures, by interconnecting cloud clusters, provide access to services across distributed infrastructures. Typically, services are replicated across clusters to ensure resilience. However, end-to-end service performance varies mainly depending on the service loads experienced by individual clusters. Therefore, a key challenge is to optimize end-to-end service performance by routing service requests to clusters with the least service processing/response times. We present a two-phase approach that combines an optimized multi-layer optical routing system with service mesh performance costs to improve end-to-end service performance. Our experimental strategy shows that leveraging a multi-layer architecture in combination with service performance information improves end-to-end performance. We evaluate our approach by testing our strategy on a service mesh layer overlay on a modified continental united states (CONUS) network topology.
17.
Alhowaidi, Mohammad; Nadig, Deepak; Hu, Boyang; Ramamurthy, Byrav; Bockelman, Brian
Cache management for large data transfers and multipath forwarding strategies in Named Data Networking Journal Article
In: Computer Networks, vol. 199, pp. 108437, 2021, ISSN: 1389-1286.
Abstract | BibTeX | Tags: Cache management, Compact Muon Solenoid, Forwarding strategy, NDN, SDN | Links:
@article{ALHOWAIDI2021108437,
title = {Cache management for large data transfers and multipath forwarding strategies in Named Data Networking},
author = {Mohammad Alhowaidi and Deepak Nadig and Boyang Hu and Byrav Ramamurthy and Brian Bockelman},
url = {https://www.sciencedirect.com/science/article/pii/S1389128621003972},
doi = {10.1016/j.comnet.2021.108437},
issn = {1389-1286},
year = {2021},
date = {2021-09-01},
urldate = {2021-09-01},
journal = {Computer Networks},
volume = {199},
pages = {108437},
abstract = {Named Data Networking (NDN) is a promising approach to provide fast in-network access to compact muon solenoid (CMS) datasets. It proposes a content-centric rather than a host-centric approach to data retrieval. Data packets with unique and immutable names are retrieved from a content store (CS) using Interest packets. The current NDN architecture relies on forwarding strategies that are only dependent upon on-path caching. Such a design does not take advantage of the cached content available on the adjacent off-path routers in the network, thus reducing data transfer efficiency. In this work, we propose a software-defined, storage-aware routing mechanism that leverages NDN router cache-states, software defined networking (SDN) and multipath forwarding strategies to improve the efficiency of very large data transfers. First, we propose a novel distributed multipath (D-MP) forwarding strategy and enhancements to the NDN Interest forwarding pipeline. In addition, we develop a centralized SDN-enabled control for the multipath forwarding strategy (S-MP), which leverages the global knowledge of NDN network states that distributes Interests efficiently. We perform extensive evaluations of our proposed methods on an at-scale wide area network (WAN) testbed spanning six geographically separated sites. Our proposed solutions easily outperform the existing NDN forwarding strategies. The D-MP strategy results in performance gains ranging between 10.4x to 12.5x over the default NDN implementation without in-network caching, and 12.2x to 18.4x with in-network caching enabled. For S-MP strategy, we demonstrate a performance improvement of 10.6x to 12.6x, and 12.9x to 18.5x, with in-network caching disabled and enabled, respectively. Further, we also present a comprehensive analysis of NDN cache management for large data transfers and propose a novel prefetching mechanism to improve data transfer performance. Due to the inherent capacity limitations of the NDN router caches, we use SDN to provide an intelligent and efficient solution for data distribution and routing across multiple NDN router caches. We demonstrate how software-defined control can be used to partition and distribute large CMS files based on NDN router cache-state knowledge. Further, SDN control will also configure the router forwarding strategy to retrieve CMS data from the network. Our proposed solution demonstrates that the CMS datasets can be retrieved 28%–38% faster from the NDN routers’ caches than existing NDN approaches. Lastly, we develop a prefetching mechanism to improve the transfer performance of files that are not available in the router’s cache.},
keywords = {Cache management, Compact Muon Solenoid, Forwarding strategy, NDN, SDN},
pubstate = {published},
tppubtype = {article}
}
Named Data Networking (NDN) is a promising approach to provide fast in-network access to compact muon solenoid (CMS) datasets. It proposes a content-centric rather than a host-centric approach to data retrieval. Data packets with unique and immutable names are retrieved from a content store (CS) using Interest packets. The current NDN architecture relies on forwarding strategies that are only dependent upon on-path caching. Such a design does not take advantage of the cached content available on the adjacent off-path routers in the network, thus reducing data transfer efficiency. In this work, we propose a software-defined, storage-aware routing mechanism that leverages NDN router cache-states, software defined networking (SDN) and multipath forwarding strategies to improve the efficiency of very large data transfers. First, we propose a novel distributed multipath (D-MP) forwarding strategy and enhancements to the NDN Interest forwarding pipeline. In addition, we develop a centralized SDN-enabled control for the multipath forwarding strategy (S-MP), which leverages the global knowledge of NDN network states that distributes Interests efficiently. We perform extensive evaluations of our proposed methods on an at-scale wide area network (WAN) testbed spanning six geographically separated sites. Our proposed solutions easily outperform the existing NDN forwarding strategies. The D-MP strategy results in performance gains ranging between 10.4x to 12.5x over the default NDN implementation without in-network caching, and 12.2x to 18.4x with in-network caching enabled. For S-MP strategy, we demonstrate a performance improvement of 10.6x to 12.6x, and 12.9x to 18.5x, with in-network caching disabled and enabled, respectively. Further, we also present a comprehensive analysis of NDN cache management for large data transfers and propose a novel prefetching mechanism to improve data transfer performance. Due to the inherent capacity limitations of the NDN router caches, we use SDN to provide an intelligent and efficient solution for data distribution and routing across multiple NDN router caches. We demonstrate how software-defined control can be used to partition and distribute large CMS files based on NDN router cache-state knowledge. Further, SDN control will also configure the router forwarding strategy to retrieve CMS data from the network. Our proposed solution demonstrates that the CMS datasets can be retrieved 28%–38% faster from the NDN routers’ caches than existing NDN approaches. Lastly, we develop a prefetching mechanism to improve the transfer performance of files that are not available in the router’s cache.
16.
Nadig, Deepak; Alaoui, Sara El; Ramamurthy, Byrav; Pitla, Santosh
ERGO: A Scalable Edge Computing Architecture for Infrastructureless Agricultural Internet of Things Proceedings Article
In: 2021 IEEE International Symposium on Local and Metropolitan Area Networks (LANMAN), pp. 1–2, 2021, (ISSN: 1944-0375).
Abstract | BibTeX | Tags: Ag-IoT, cloud computing, Computer architecture, Edge Computing, Infrastructureless, Instruments, machine learning, Metropolitan area networks, Performance evaluation, Throughput | Links:
@inproceedings{nadig_ergo_2021,
title = {ERGO: A Scalable Edge Computing Architecture for Infrastructureless Agricultural Internet of Things},
author = {Deepak Nadig and Sara El Alaoui and Byrav Ramamurthy and Santosh Pitla},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/Nadig-et-al.-2021-ERGO-A-Scalable-Edge-Computing-Architecture-for-I.pdf},
doi = {10.1109/LANMAN52105.2021.9478811},
year = {2021},
date = {2021-07-01},
urldate = {2021-07-01},
booktitle = {2021 IEEE International Symposium on Local and Metropolitan Area Networks (LANMAN)},
pages = {1--2},
abstract = {In this paper, we propose ERGO (edge architecture for Ag-IoT), an edge-computing architecture for infrastructureless smart agriculture environments. We also develop Ag-IoT application APIs and the associated microservice infrastructure. Our implementation and evaluations show that ERGO can operate independently of cloud-backed assistance, is highly scalable, modular, and affords composability benefits to Ag-IoT systems. We also demonstrate that ERGO outperforms traditional infrastructure in response latencies and transactional throughput, on average, by over 54% and 77%, respectively.},
note = {ISSN: 1944-0375},
keywords = {Ag-IoT, cloud computing, Computer architecture, Edge Computing, Infrastructureless, Instruments, machine learning, Metropolitan area networks, Performance evaluation, Throughput},
pubstate = {published},
tppubtype = {inproceedings}
}
In this paper, we propose ERGO (edge architecture for Ag-IoT), an edge-computing architecture for infrastructureless smart agriculture environments. We also develop Ag-IoT application APIs and the associated microservice infrastructure. Our implementation and evaluations show that ERGO can operate independently of cloud-backed assistance, is highly scalable, modular, and affords composability benefits to Ag-IoT systems. We also demonstrate that ERGO outperforms traditional infrastructure in response latencies and transactional throughput, on average, by over 54% and 77%, respectively.
15.
Nadig, Deepak
2021, ISBN: 9798534693966.
Abstract | BibTeX | Tags: | Links:
@phdthesis{nokey,
title = {Application-Awareness in Softwarized Networks: Building Intelligent Networks through Application and Network-Layer Collaboration},
author = {Deepak Nadig},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/dissertation.pdf},
isbn = {9798534693966},
year = {2021},
date = {2021-07-01},
urldate = {2021-01-01},
journal = {ProQuest Dissertations and Theses},
pages = {223},
abstract = {Increasingly, campus networks manage a multitude of large-scale data transfers. Big data plays a pivotal role in university research and impacts engineering, agriculture, natural sciences, and humanities. Campus network infrastructures support multiple network management goals, including commodity internet traffic and high-performance networks for scientific research. These goals often impose conflicting requirements on network design and management, and therefore, networks optimized and specially engineered for data-intensive tasks are necessary. Further, many aspects of campus networks are hard to change without impacting regular network operation. Over the years, numerous solutions have focused on the management and security of large-scale data transfers. These solutions severely degrade data transfer performance or result in data flows completely bypassing the campus network management and security controls.This dissertation will study application-aware architectures and present software defined networking (SDN) and network functions virtualization (NFV) solutions for data-intensive science. Our proposed application-aware SDN solutions span network monitoring, management, service differentiation, and security for data-intensive applications. We first propose a novel application-aware architecture called SNAG (SDN-managed Network Architecture for GridFTP transfers). SNAG combines application-awareness with SDN-enabled network management to classify, monitor and manage network resources actively. At HCC, we also demonstrate how our system ensures the quality of service (QoS) for high-throughput workflows such as Compact Muon Solenoid (CMS) and Laser Interferometer Gravitational-Wave Observatory (LIGO). Next, we develop a novel application-aware flow reduction (AAFR) algorithm to optimally map service function chains (SFC) to multiple data centers while adhering to the data center’s capacity constraints. We then present an application-aware intelligent load balancing system for high-throughput, distributed computing workflows. Our solution integrates with a major U.S. CMS Tier-2 site. Lastly, by developing a scalable application-aware edge computing framework, we focus on building reliable service-to-service communication across distributed infrastructures using a service mesh architecture. By building application-aware architectures and evolving data-intensive applications to collaboratively and securely share application-layer metadata with the network-layer, we pave the way for intelligent networks that are secure, automated, dynamically composable and highly scalable.},
keywords = {},
pubstate = {published},
tppubtype = {phdthesis}
}
Increasingly, campus networks manage a multitude of large-scale data transfers. Big data plays a pivotal role in university research and impacts engineering, agriculture, natural sciences, and humanities. Campus network infrastructures support multiple network management goals, including commodity internet traffic and high-performance networks for scientific research. These goals often impose conflicting requirements on network design and management, and therefore, networks optimized and specially engineered for data-intensive tasks are necessary. Further, many aspects of campus networks are hard to change without impacting regular network operation. Over the years, numerous solutions have focused on the management and security of large-scale data transfers. These solutions severely degrade data transfer performance or result in data flows completely bypassing the campus network management and security controls.This dissertation will study application-aware architectures and present software defined networking (SDN) and network functions virtualization (NFV) solutions for data-intensive science. Our proposed application-aware SDN solutions span network monitoring, management, service differentiation, and security for data-intensive applications. We first propose a novel application-aware architecture called SNAG (SDN-managed Network Architecture for GridFTP transfers). SNAG combines application-awareness with SDN-enabled network management to classify, monitor and manage network resources actively. At HCC, we also demonstrate how our system ensures the quality of service (QoS) for high-throughput workflows such as Compact Muon Solenoid (CMS) and Laser Interferometer Gravitational-Wave Observatory (LIGO). Next, we develop a novel application-aware flow reduction (AAFR) algorithm to optimally map service function chains (SFC) to multiple data centers while adhering to the data center’s capacity constraints. We then present an application-aware intelligent load balancing system for high-throughput, distributed computing workflows. Our solution integrates with a major U.S. CMS Tier-2 site. Lastly, by developing a scalable application-aware edge computing framework, we focus on building reliable service-to-service communication across distributed infrastructures using a service mesh architecture. By building application-aware architectures and evolving data-intensive applications to collaboratively and securely share application-layer metadata with the network-layer, we pave the way for intelligent networks that are secure, automated, dynamically composable and highly scalable.
2020
14.
Nadig, Deepak; Alaoui, Sara El; Ramamurthy, Byrav
ERGO: A Scalable Edge Computing Architecture for Ag-IoT Proceedings Article
In: 3rd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 20), USENIX Association, 2020.
BibTeX | Tags: | Links:
@inproceedings{nadig_ergo_2020,
title = {ERGO: A Scalable Edge Computing Architecture for Ag-IoT},
author = {Deepak Nadig and Sara El Alaoui and Byrav Ramamurthy},
url = {https://www.usenix.org/conference/hotedge20/presentation/nadig},
year = {2020},
date = {2020-06-01},
urldate = {2020-06-01},
booktitle = {3rd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 20)},
publisher = {USENIX Association},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2019
13.
Nayak, Sampashree; Nadig, Deepak; Ramamurthy, Byrav
Analyzing Malicious URLs using a Threat Intelligence System Proceedings Article
In: 2019 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), pp. 1–4, 2019, (ISSN: 2153-1684).
Abstract | BibTeX | Tags: k-means clustering, machine learning, threat intelligence feeds, URL analysis | Links:
@inproceedings{nayak_analyzing_2019,
title = {Analyzing Malicious URLs using a Threat Intelligence System},
author = {Sampashree Nayak and Deepak Nadig and Byrav Ramamurthy},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/Nayak-et-al.-2019-Analyzing-Malicious-URLs-using-a-Threat-Intelligen.pdf},
doi = {10.1109/ANTS47819.2019.9118051},
year = {2019},
date = {2019-12-01},
urldate = {2019-12-01},
booktitle = {2019 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS)},
pages = {1--4},
abstract = {Threat intelligence and management systems form a vital component of an organization's cybersecurity infrastructure. Threat intelligence, when used with active monitoring of network traffic, can be critical to ensure reliable data communication between endpoints. Threat intelligence systems are well suited for analyzing anomalous behaviors in network traffic and can be employed to assist organizations in identifying and successfully responding to cyber-attacks. In this paper, we present a machine learning approach for clustering malicious uniform resource locators (URLs). We focus on a URL dataset gathered from a threat intelligence feeds framework. We implement a k-means clustering solution for grouping malicious URLs obtained from open source threat intelligence feeds. We demonstrate the effectiveness of our unsupervised learning technique to discover the hidden structures in the malicious URL dataset. Our URL keyword/text clustering solution provides valuable insights about the malicious URLs and aids network operators in policy decisions to mitigate cyber-attacks. The clusters obtained using our approach has a silhouette coefficient of 0.383 for a dataset containing over 11,000 malicious URLs. Lastly, we develop a probabilistic scoring model to calculate the percentage of malicious keywords present in a given URL. After analyzing over 72,000 malicious keywords, our model successfully identifies over 80% of the URLs in a test dataset as malicious.},
note = {ISSN: 2153-1684},
keywords = {k-means clustering, machine learning, threat intelligence feeds, URL analysis},
pubstate = {published},
tppubtype = {inproceedings}
}
Threat intelligence and management systems form a vital component of an organization's cybersecurity infrastructure. Threat intelligence, when used with active monitoring of network traffic, can be critical to ensure reliable data communication between endpoints. Threat intelligence systems are well suited for analyzing anomalous behaviors in network traffic and can be employed to assist organizations in identifying and successfully responding to cyber-attacks. In this paper, we present a machine learning approach for clustering malicious uniform resource locators (URLs). We focus on a URL dataset gathered from a threat intelligence feeds framework. We implement a k-means clustering solution for grouping malicious URLs obtained from open source threat intelligence feeds. We demonstrate the effectiveness of our unsupervised learning technique to discover the hidden structures in the malicious URL dataset. Our URL keyword/text clustering solution provides valuable insights about the malicious URLs and aids network operators in policy decisions to mitigate cyber-attacks. The clusters obtained using our approach has a silhouette coefficient of 0.383 for a dataset containing over 11,000 malicious URLs. Lastly, we develop a probabilistic scoring model to calculate the percentage of malicious keywords present in a given URL. After analyzing over 72,000 malicious keywords, our model successfully identifies over 80% of the URLs in a test dataset as malicious.
12.
Alhowaidi, Mohammad; Nadig, Deepak; Ramamurthy, Byrav
Cache Management for Large Data Transfers in Named Data Networking using SDN Proceedings Article
In: 2019 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), pp. 1–6, 2019, (ISSN: 2153-1684).
Abstract | BibTeX | Tags: | Links:
@inproceedings{alhowaidi_cache_2019,
title = {Cache Management for Large Data Transfers in Named Data Networking using SDN},
author = {Mohammad Alhowaidi and Deepak Nadig and Byrav Ramamurthy},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/Alhowaidi-et-al.-2019-Cache-Management-for-Large-Data-Transfers-in-Named.pdf},
doi = {10.1109/ANTS47819.2019.9118137},
year = {2019},
date = {2019-12-01},
urldate = {2019-12-01},
booktitle = {2019 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS)},
pages = {1--6},
abstract = {The Compact Muon Solenoid (CMS) on the Large Hadron Collider (LHC) manage high volumes of data that currently exceeds 100PB across different sites. An important challenge of delivering data to experimenters in the CMS workflow is the data volume. An experiment data file has an average size of 2 Gigabytes, with file sizes ranging between 100 Megabytes and 20 Gigabytes. Also, a complete dataset comprises of multiple files, with the dataset files ranging from 2 Terabytes and 100 Terabytes in size. Providing fast access to datasets is an important enabler for data-intensive science research. In our work, we demonstrate a Information-Centric Networking (ICN) approach to providing fast in-network access to CMS datasets. To that end, we must first address the problem of how to store large CMS files in network caches closer to the end-users. We propose a software-defined, storage-aware routing mechanism using named data networking (NDN) to achieve this goal. Due to the inherent capacity limitations of the NDN router caches, we use software defined networking (SDN) to provide an intelligent and efficient solution for data distribution and routing across multiple NDN router caches. We demonstrate how software-defined control can be used for partitioning and distributing large CMS files based on NDN router cache-state knowledge. Further, SDN control will also configure the router forwarding strategy to retrieve CMS data from the network. Using our proposed architecture, we show that CMS dataset can be retrieved 28%–38% faster from the NDN routers caches compared to existing approaches. Lastly, we develop a prefetching mechanism to improve the transfer performance of files not available in the router's cache.},
note = {ISSN: 2153-1684},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
The Compact Muon Solenoid (CMS) on the Large Hadron Collider (LHC) manage high volumes of data that currently exceeds 100PB across different sites. An important challenge of delivering data to experimenters in the CMS workflow is the data volume. An experiment data file has an average size of 2 Gigabytes, with file sizes ranging between 100 Megabytes and 20 Gigabytes. Also, a complete dataset comprises of multiple files, with the dataset files ranging from 2 Terabytes and 100 Terabytes in size. Providing fast access to datasets is an important enabler for data-intensive science research. In our work, we demonstrate a Information-Centric Networking (ICN) approach to providing fast in-network access to CMS datasets. To that end, we must first address the problem of how to store large CMS files in network caches closer to the end-users. We propose a software-defined, storage-aware routing mechanism using named data networking (NDN) to achieve this goal. Due to the inherent capacity limitations of the NDN router caches, we use software defined networking (SDN) to provide an intelligent and efficient solution for data distribution and routing across multiple NDN router caches. We demonstrate how software-defined control can be used for partitioning and distributing large CMS files based on NDN router cache-state knowledge. Further, SDN control will also configure the router forwarding strategy to retrieve CMS data from the network. Using our proposed architecture, we show that CMS dataset can be retrieved 28%–38% faster from the NDN routers caches compared to existing approaches. Lastly, we develop a prefetching mechanism to improve the transfer performance of files not available in the router's cache.
11.
Nadig, D.; Ramamurthy, B.; Bockelman, B.; Swanson, D.
APRIL: An Application-Aware, Predictive and Intelligent Load Balancing Solution for Data-Intensive Science Proceedings Article
In: IEEE INFOCOM 2019 - IEEE Conference on Computer Communications, pp. 1909–1917, 2019.
Abstract | BibTeX | Tags: Big Data, Correlation, Deep learning, Load management, Load modeling, Predictive models, Servers | Links:
@inproceedings{nadig_april:_2019,
title = {APRIL: An Application-Aware, Predictive and Intelligent Load Balancing Solution for Data-Intensive Science},
author = {D. Nadig and B. Ramamurthy and B. Bockelman and D. Swanson},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/Nadig-et-al.-2019-APRIL-An-Application-Aware-Predictive-and-Intell.pdf},
doi = {10.1109/INFOCOM.2019.8737537},
year = {2019},
date = {2019-04-01},
urldate = {2019-04-01},
booktitle = {IEEE INFOCOM 2019 - IEEE Conference on Computer Communications},
pages = {1909--1917},
abstract = {In this paper, we propose an application-aware intelligent load balancing system for high-throughput, distributed computing, and data-intensive science workflows. We leverage emerging deep learning techniques for time-series modeling to develop an application-aware predictive analytics system for accurately forecasting GridFTP connection loads. Our solution integrates with a major U.S. CMS Tier-2 site; we use a real dataset representing 670 million GridFTP transfer connections measured over 18 months to drive our predictive analytics solution. First, we perform extensive analysis on this dataset and use the connection loads as an example to study the temporal dependencies between various user-roles and workflow memberships. We use the analysis to motivate the design of a gated recurrent unit (GRU) based deep recurrent neural network (RNN) for modeling long-term temporal dependencies and predicting connection loads. We develop a novel application-aware, predictive and intelligent load balancer, APRIL, that effectively integrates application metadata and load forecast information to maximize server utilization. We conduct extensive experiments to evaluate the performance of our deep RNN predictive analytics system and compare it with other approaches such as ARIMA and multi-layer perceptron (MLP) predictors. The results show that our forecasting model, depending on the user-role, performs between 5.88%–92.6% better than the alternatives. We also demonstrate the effectiveness of APRIL by comparing it with the load balancing capabilities of an existing production Linux Virtual Server (LVS) cluster. Our approach improves server utilization, on an average, between 0.5 to 11 times, when compared with its LVS counterpart.},
keywords = {Big Data, Correlation, Deep learning, Load management, Load modeling, Predictive models, Servers},
pubstate = {published},
tppubtype = {inproceedings}
}
In this paper, we propose an application-aware intelligent load balancing system for high-throughput, distributed computing, and data-intensive science workflows. We leverage emerging deep learning techniques for time-series modeling to develop an application-aware predictive analytics system for accurately forecasting GridFTP connection loads. Our solution integrates with a major U.S. CMS Tier-2 site; we use a real dataset representing 670 million GridFTP transfer connections measured over 18 months to drive our predictive analytics solution. First, we perform extensive analysis on this dataset and use the connection loads as an example to study the temporal dependencies between various user-roles and workflow memberships. We use the analysis to motivate the design of a gated recurrent unit (GRU) based deep recurrent neural network (RNN) for modeling long-term temporal dependencies and predicting connection loads. We develop a novel application-aware, predictive and intelligent load balancer, APRIL, that effectively integrates application metadata and load forecast information to maximize server utilization. We conduct extensive experiments to evaluate the performance of our deep RNN predictive analytics system and compare it with other approaches such as ARIMA and multi-layer perceptron (MLP) predictors. The results show that our forecasting model, depending on the user-role, performs between 5.88%–92.6% better than the alternatives. We also demonstrate the effectiveness of APRIL by comparing it with the load balancing capabilities of an existing production Linux Virtual Server (LVS) cluster. Our approach improves server utilization, on an average, between 0.5 to 11 times, when compared with its LVS counterpart.
10.
Nadig, Deepak; Ramamurthy, Byrav
Securing Large-scale Data Transfers in Campus Networks: Experiences, Issues, and Challenges Proceedings Article
In: Proceedings of the ACM International Workshop on Security in Software Defined Networks & Network Function Virtualization, pp. 29–32, ACM, New York, NY, USA, 2019, ISBN: 978-1-4503-6179-8, (event-place: Richardson, Texas, USA).
Abstract | BibTeX | Tags: application-awareness, data-intensive science, network functions virtualization, security, software defined networks | Links:
@inproceedings{nadig_securing_2019,
title = {Securing Large-scale Data Transfers in Campus Networks: Experiences, Issues, and Challenges},
author = {Deepak Nadig and Byrav Ramamurthy},
url = {http://doi.acm.org/10.1145/3309194.3309444},
doi = {10.1145/3309194.3309444},
isbn = {978-1-4503-6179-8},
year = {2019},
date = {2019-01-01},
urldate = {2019-03-28},
booktitle = {Proceedings of the ACM International Workshop on Security in Software Defined Networks & Network Function Virtualization},
pages = {29--32},
publisher = {ACM},
address = {New York, NY, USA},
series = {SDN-NFVSec '19},
abstract = {Increasingly, campus networks manage a multitude of large-scale data transfers. Big data plays a pivotal role in university research and impacts domains such as engineering, agriculture, natural sciences, and humanities. Over the years, numerous solutions have been proposed to manage and secure large-scale data transfers efficiently. Examples consist of the inclusion of security policies at the network edge, optimized middlebox management, and the Science Demilitarized Zone (Science DMZ). These solutions either severely degrade data transfer performance or result in data flows completely bypassing the campus network security controls. In this paper, we present our experience with the design, development, and management of large-scale data transfers using software defined networking (SDN) and network functions virtualization (NFV). We also discuss the issues and challenges associated with securing large-scale data transfers in campus networks.},
note = {event-place: Richardson, Texas, USA},
keywords = {application-awareness, data-intensive science, network functions virtualization, security, software defined networks},
pubstate = {published},
tppubtype = {inproceedings}
}
Increasingly, campus networks manage a multitude of large-scale data transfers. Big data plays a pivotal role in university research and impacts domains such as engineering, agriculture, natural sciences, and humanities. Over the years, numerous solutions have been proposed to manage and secure large-scale data transfers efficiently. Examples consist of the inclusion of security policies at the network edge, optimized middlebox management, and the Science Demilitarized Zone (Science DMZ). These solutions either severely degrade data transfer performance or result in data flows completely bypassing the campus network security controls. In this paper, we present our experience with the design, development, and management of large-scale data transfers using software defined networking (SDN) and network functions virtualization (NFV). We also discuss the issues and challenges associated with securing large-scale data transfers in campus networks.
2018
9.
Alhowaidi, M.; Nadig, D.; Ramamurthy, B.; Bockelman, B.; Swanson, D.
Multipath Forwarding Strategies and SDN Control for Named Data Networking Proceedings Article
In: 2018 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), pp. 1–6, 2018.
Abstract | BibTeX | Tags: Computer architecture, Data transfer, IP networks, Pipeline processing, Pipelines, Routing, Routing protocols | Links:
@inproceedings{alhowaidi_multipath_2018,
title = {Multipath Forwarding Strategies and SDN Control for Named Data Networking},
author = {M. Alhowaidi and D. Nadig and B. Ramamurthy and B. Bockelman and D. Swanson},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/Alhowaidi-et-al.-2018-Multipath-Forwarding-Strategies-and-SDN-Control-fo.pdf},
doi = {10.1109/ANTS.2018.8710068},
year = {2018},
date = {2018-12-01},
urldate = {2018-12-01},
booktitle = {2018 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS)},
pages = {1--6},
abstract = {Named Data Networking (NDN) proposes a contentcentric rather than a host-centric approach to data retrieval. Data packets with unique and immutable names are retrieved from a content store (CS) using Interest packets. The current NDN architecture relies on forwarding strategies that are dependent upon on-path caching and is therefore inefficient. This approach reduces data transfer efficiency by ignoring the cached content available on the adjacent off-path routers in the network. In this paper, we propose a novel distributed multipath (D-MP) forwarding strategy and enhancements to the NDN Interest forwarding pipeline. Furthermore, we develop a centralized SDNenabled control for the multipath forwarding strategy (S-MP) that distributes Interests efficiently by using the global knowledge of the NDN network states. We perform extensive evaluations of our proposed methods on an at-scale WAN testbed spanning six geographically separated sites. Our solutions outperform the existing NDN forwarding strategies by a significant margin. We show that the D-MP strategy results in performance gains ranging between 10.4x to 12.5x over the default NDN implementation without in-network caching, and gains of 12.2x to 18.4x with in-network caching. In addition, for the S-MP case, we demonstrate a performance improvement of 10.6x to 12.6x, and 12.9x to 18.5x, for with- and without in-network caching respectively.},
keywords = {Computer architecture, Data transfer, IP networks, Pipeline processing, Pipelines, Routing, Routing protocols},
pubstate = {published},
tppubtype = {inproceedings}
}
Named Data Networking (NDN) proposes a contentcentric rather than a host-centric approach to data retrieval. Data packets with unique and immutable names are retrieved from a content store (CS) using Interest packets. The current NDN architecture relies on forwarding strategies that are dependent upon on-path caching and is therefore inefficient. This approach reduces data transfer efficiency by ignoring the cached content available on the adjacent off-path routers in the network. In this paper, we propose a novel distributed multipath (D-MP) forwarding strategy and enhancements to the NDN Interest forwarding pipeline. Furthermore, we develop a centralized SDNenabled control for the multipath forwarding strategy (S-MP) that distributes Interests efficiently by using the global knowledge of the NDN network states. We perform extensive evaluations of our proposed methods on an at-scale WAN testbed spanning six geographically separated sites. Our solutions outperform the existing NDN forwarding strategies by a significant margin. We show that the D-MP strategy results in performance gains ranging between 10.4x to 12.5x over the default NDN implementation without in-network caching, and gains of 12.2x to 18.4x with in-network caching. In addition, for the S-MP case, we demonstrate a performance improvement of 10.6x to 12.6x, and 12.9x to 18.5x, for with- and without in-network caching respectively.
8.
Nadig, D.; Jung, E.; Kettimuthu, R.; Foster, I.; Rao, S. V. Nageswara; Ramamurthy, B.
Comparative Performance Evaluation of High-performance Data Transfer Tools Proceedings Article
In: 2018 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), pp. 1–6, 2018.
Abstract | BibTeX | Tags: Data transfer, Local area networks, Protocols, Reliability, Software architecture, Tools, Wide area networks | Links:
@inproceedings{nadig_comparative_2018,
title = {Comparative Performance Evaluation of High-performance Data Transfer Tools},
author = {D. Nadig and E. Jung and R. Kettimuthu and I. Foster and S. V. Nageswara Rao and B. Ramamurthy},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/Nadig-et-al.-2018-Comparative-Performance-Evaluation-of-High-perform.pdf},
doi = {10.1109/ANTS.2018.8710071},
year = {2018},
date = {2018-12-01},
urldate = {2018-12-01},
booktitle = {2018 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS)},
pages = {1--6},
abstract = {Data transfer in wide-area networks has been long studied in different contexts, from data sharing among data centers to online access to scientific data. Many software tools and platforms have been developed to facilitate easy, reliable, fast, and secure data transfer over wide area networks, such as GridFTP, FDT, bbcp, mdtmFTP, and XDD. However, few studies have shown the full capabilities of existing data transfer tools from the perspective of whether such tools have fully adopted state-of-the-art techniques through meticulous comparative evaluations. In this paper, we evaluate the performance of the four highperformance data transfer tools (GridFTP, FDT, mdtmFTP, and XDD) in various environments. Our evaluation suggests that each tool has strengths and weaknesses. FDT and GridFTP perform consistently in diverse environments. XDD and mdtmFTP show improved performance in limited environments and datasets during our evaluation. Unlike other studies on data transfer tools, we also evaluate the predictability of the tools’ performance, an important factor for scheduling different stages of science workflows. Performance predictability also helps in (auto)tuning the configurable parameters of the data transfer tool. We apply statistical learning techniques such as linear/polynomial regression, and k-nearest neighbors (kNN), to assess the performance predictability of each tool using its control parameters. Our results show that we can achieve good prediction performance for GridFTP and mdtmFTP using linear regression and kNN, respectively.},
keywords = {Data transfer, Local area networks, Protocols, Reliability, Software architecture, Tools, Wide area networks},
pubstate = {published},
tppubtype = {inproceedings}
}
Data transfer in wide-area networks has been long studied in different contexts, from data sharing among data centers to online access to scientific data. Many software tools and platforms have been developed to facilitate easy, reliable, fast, and secure data transfer over wide area networks, such as GridFTP, FDT, bbcp, mdtmFTP, and XDD. However, few studies have shown the full capabilities of existing data transfer tools from the perspective of whether such tools have fully adopted state-of-the-art techniques through meticulous comparative evaluations. In this paper, we evaluate the performance of the four highperformance data transfer tools (GridFTP, FDT, mdtmFTP, and XDD) in various environments. Our evaluation suggests that each tool has strengths and weaknesses. FDT and GridFTP perform consistently in diverse environments. XDD and mdtmFTP show improved performance in limited environments and datasets during our evaluation. Unlike other studies on data transfer tools, we also evaluate the predictability of the tools’ performance, an important factor for scheduling different stages of science workflows. Performance predictability also helps in (auto)tuning the configurable parameters of the data transfer tool. We apply statistical learning techniques such as linear/polynomial regression, and k-nearest neighbors (kNN), to assess the performance predictability of each tool using its control parameters. Our results show that we can achieve good prediction performance for GridFTP and mdtmFTP using linear regression and kNN, respectively.
7.
Nadig, D.; Ramamurthy, B.; Bockelman, B.; Swanson, D.
Large Data Transfer Predictability and Forecasting using Application-Aware SDN Proceedings Article
In: 2018 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), pp. 1–6, 2018.
Abstract | BibTeX | Tags: Aggregates, Analytical models, Data analysis, Data models, Data transfer, Forecasting, Predictive models | Links:
@inproceedings{nadig_large_2018,
title = {Large Data Transfer Predictability and Forecasting using Application-Aware SDN},
author = {D. Nadig and B. Ramamurthy and B. Bockelman and D. Swanson},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/Nadig-et-al.-2018-Large-Data-Transfer-Predictability-and-Forecasting.pdf},
doi = {10.1109/ANTS.2018.8710165},
year = {2018},
date = {2018-12-01},
urldate = {2018-12-01},
booktitle = {2018 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS)},
pages = {1--6},
abstract = {Network management for applications that rely on large-scale data transfers is challenging due to the volatility and the dynamic nature of the access traffic patterns. Predictive analytics and forecasting play an important role in providing effective resource allocation strategies for large data transfers. We propose a predictive analytics solution for large data transfers using an application-aware software defined networking (SDN) approach. We perform extensive exploratory data analysis to characterize the GridFTP connection transfers dataset and present various strategies for its use with statistical forecasting models. We develop a univariate autoregressive integrated moving average (ARIMA) based prediction framework for forecasting GridFTP connection transfers. Our prediction model tightly integrates with an application-aware SDN solution to preemptively drive network management decisions for GridFTP resource allocation at a U.S. CMS Tier-2 site. Further, our framework has a mean absolute percentage error (MAPE) ranging from 6% to 10% when applied to make rolling forecasts.},
keywords = {Aggregates, Analytical models, Data analysis, Data models, Data transfer, Forecasting, Predictive models},
pubstate = {published},
tppubtype = {inproceedings}
}
Network management for applications that rely on large-scale data transfers is challenging due to the volatility and the dynamic nature of the access traffic patterns. Predictive analytics and forecasting play an important role in providing effective resource allocation strategies for large data transfers. We propose a predictive analytics solution for large data transfers using an application-aware software defined networking (SDN) approach. We perform extensive exploratory data analysis to characterize the GridFTP connection transfers dataset and present various strategies for its use with statistical forecasting models. We develop a univariate autoregressive integrated moving average (ARIMA) based prediction framework for forecasting GridFTP connection transfers. Our prediction model tightly integrates with an application-aware SDN solution to preemptively drive network management decisions for GridFTP resource allocation at a U.S. CMS Tier-2 site. Further, our framework has a mean absolute percentage error (MAPE) ranging from 6% to 10% when applied to make rolling forecasts.
6.
Nadig, D.; Ramamurthy, B.; Bockelman, B.; Swanson, D.
Optimized Service Chain Mapping and reduced flow processing with Application-Awareness Proceedings Article
In: 2018 4th IEEE Conference on Network Softwarization and Workshops (NetSoft), pp. 303–307, 2018.
Abstract | BibTeX | Tags: AAFR algorithm, application-aware flow reduction algorithm, application-awareness, Bandwidth, capacitated-SFC mapping case, capacitated/uncapacitated flow, cloud computing, commercial-off-the-shelf hardware, computer centres, Conferences, cost gains, data center, Data centers, Data models, flow processing cost, flow-processing costs, flow-to-SFC mappings, integer linear programming formulation, integer programming, Internet, linear programming, multiple data centers, network function virtualization, network functions, network functions virtualization, optimally map service function chains, optimized service chain mapping, security, Service Chaining, service function chains, SFC mapping, SFC mapping problem, SFC-ILP, software defined networks, Substrates, virtualisation, virtualized services | Links:
@inproceedings{nadig_optimized_2018,
title = {Optimized Service Chain Mapping and reduced flow processing with Application-Awareness},
author = {D. Nadig and B. Ramamurthy and B. Bockelman and D. Swanson},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/Nadig-et-al.-2018-Optimized-Service-Chain-Mapping-and-reduced-flow-p.pdf},
doi = {10.1109/NETSOFT.2018.8459912},
year = {2018},
date = {2018-06-01},
urldate = {2018-06-01},
booktitle = {2018 4th IEEE Conference on Network Softwarization and Workshops (NetSoft)},
pages = {303--307},
abstract = {Network Function Virtualization (NFV) brings a new set of challenges when deploying virtualized services on commercial-off-the-shelf (COTS) hardware. Network functions can be dynamically managed to provide the necessary services on-demand and further, services can be chained together to form a larger composite. In this paper, we address an important technical problem of mapping service function chains (SFCs) across different data centers with the objective of reducing the flow processing costs. We develop an integer linear programming (ILP) formulation to optimally map service function chains to multiple data centers while adhering to the data center's capacity constraints. We propose a novel application-aware flow reduction (AAFR) algorithm to simplify the SFC-ILP to significantly reduce the number of flows processed by the SFCs. We perform a thorough study of the SFC mapping problem for multiple data centers and evaluate the performance of our proposed approach with respect to three parameters: i) impact of number of SFCs and SFC length on flow processing cost, ii) capacitated/uncapacitated flow processing cost gains, and iii) balancing flow-to-SFC mappings across data centers. Our evaluations show that our proposed AAFR algorithm reduces flow-processing costs by 70% for the capacitated-SFC mapping case over the SFC-ILP. In addition, our uncapacitated AAFR (AAFR-U) algorithm provides a further 4.1% cost-gain over its capacitated counterpart (AAFR-C).},
keywords = {AAFR algorithm, application-aware flow reduction algorithm, application-awareness, Bandwidth, capacitated-SFC mapping case, capacitated/uncapacitated flow, cloud computing, commercial-off-the-shelf hardware, computer centres, Conferences, cost gains, data center, Data centers, Data models, flow processing cost, flow-processing costs, flow-to-SFC mappings, integer linear programming formulation, integer programming, Internet, linear programming, multiple data centers, network function virtualization, network functions, network functions virtualization, optimally map service function chains, optimized service chain mapping, security, Service Chaining, service function chains, SFC mapping, SFC mapping problem, SFC-ILP, software defined networks, Substrates, virtualisation, virtualized services},
pubstate = {published},
tppubtype = {inproceedings}
}
Network Function Virtualization (NFV) brings a new set of challenges when deploying virtualized services on commercial-off-the-shelf (COTS) hardware. Network functions can be dynamically managed to provide the necessary services on-demand and further, services can be chained together to form a larger composite. In this paper, we address an important technical problem of mapping service function chains (SFCs) across different data centers with the objective of reducing the flow processing costs. We develop an integer linear programming (ILP) formulation to optimally map service function chains to multiple data centers while adhering to the data center's capacity constraints. We propose a novel application-aware flow reduction (AAFR) algorithm to simplify the SFC-ILP to significantly reduce the number of flows processed by the SFCs. We perform a thorough study of the SFC mapping problem for multiple data centers and evaluate the performance of our proposed approach with respect to three parameters: i) impact of number of SFCs and SFC length on flow processing cost, ii) capacitated/uncapacitated flow processing cost gains, and iii) balancing flow-to-SFC mappings across data centers. Our evaluations show that our proposed AAFR algorithm reduces flow-processing costs by 70% for the capacitated-SFC mapping case over the SFC-ILP. In addition, our uncapacitated AAFR (AAFR-U) algorithm provides a further 4.1% cost-gain over its capacitated counterpart (AAFR-C).
5.
Nadig, Deepak; Ramamurthy, Byrav; Bockelman, Brian; Swanson, David
Identifying Anomalies in GridFTP Transfers for Data-Intensive Science Through Application-Awareness Proceedings Article
In: Proceedings of the 2018 ACM International Workshop on Security in Software Defined Networks & Network Function Virtualization, pp. 7–12, ACM, New York, NY, USA, 2018, ISBN: 978-1-4503-5635-0, (event-place: Tempe, AZ, USA).
Abstract | BibTeX | Tags: anomaly detection, application-awareness, gridftp, software defined networks. | Links:
@inproceedings{nadig_identifying_2018,
title = {Identifying Anomalies in GridFTP Transfers for Data-Intensive Science Through Application-Awareness},
author = {Deepak Nadig and Byrav Ramamurthy and Brian Bockelman and David Swanson},
url = {http://doi.acm.org/10.1145/3180465.3180469},
doi = {10.1145/3180465.3180469},
isbn = {978-1-4503-5635-0},
year = {2018},
date = {2018-01-01},
urldate = {2019-02-07},
booktitle = {Proceedings of the 2018 ACM International Workshop on Security in Software Defined Networks & Network Function Virtualization},
pages = {7--12},
publisher = {ACM},
address = {New York, NY, USA},
series = {SDN-NFV Sec'18},
abstract = {Network anomaly detection systems can be used to identify anomalous transfers or threats, which, when undetected, can trigger large-scale malicious events. Data-intensive science projects rely on high-throughput computing and high-speed networking resources for data analysis and processing. In this paper, we propose an anomaly detection framework and architecture for identifying anomalies in GridFTP transfers. Application-awareness plays an important role in our proposed architecture and is used to communicate GridFTP application metadata to the machine learning and anomaly detection system. We demonstrate the effectiveness of our architecture by evaluating the framework with a real-world, large-scale dataset of GridFTP transfers. Preliminary results show that our framework can be used to develop novel anomaly detection services with diverse feature sets for distributed and data-intensive projects.},
note = {event-place: Tempe, AZ, USA},
keywords = {anomaly detection, application-awareness, gridftp, software defined networks.},
pubstate = {published},
tppubtype = {inproceedings}
}
Network anomaly detection systems can be used to identify anomalous transfers or threats, which, when undetected, can trigger large-scale malicious events. Data-intensive science projects rely on high-throughput computing and high-speed networking resources for data analysis and processing. In this paper, we propose an anomaly detection framework and architecture for identifying anomalies in GridFTP transfers. Application-awareness plays an important role in our proposed architecture and is used to communicate GridFTP application metadata to the machine learning and anomaly detection system. We demonstrate the effectiveness of our architecture by evaluating the framework with a real-world, large-scale dataset of GridFTP transfers. Preliminary results show that our framework can be used to develop novel anomaly detection services with diverse feature sets for distributed and data-intensive projects.
2017
4.
Nadig, D.; Ramamurthy, B.; Bockelman, B.; Swanson, D.
Differentiated network services for data-intensive science using application-aware SDN Best Paper Proceedings Article
In: 2017 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), pp. 1–6, 2017.
Abstract | BibTeX | Tags: application-aware SDN, application-aware software-defined networking, application-awareness, Compact Muon Solenoid, Cryptography, Data transfer, data-intensive science, data-intensive science projects, differentiated network services, DiffServ networks, Engines, fault-tolerant protocols, gravitational wave detectors, gridftp, GridFTP protocol, high-delay wide area network, high-energy physics projects, Laser Interferometer Gravitational-Wave Observatory, Metadata, physics computing, policy-driven approach, Protocols, queueing theory, queuing system, Servers, software defined networking, software defined networks, Wide area networks | Links:
@inproceedings{nadig_differentiated_2017,
title = {Differentiated network services for data-intensive science using application-aware SDN},
author = {D. Nadig and B. Ramamurthy and B. Bockelman and D. Swanson},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/Anantha-et-al.-2017-Differentiated-network-services-for-data-intensive.pdf},
doi = {10.1109/ANTS.2017.8384105},
year = {2017},
date = {2017-12-01},
urldate = {2017-12-01},
booktitle = {2017 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS)},
pages = {1--6},
abstract = {Data-intensive science projects rely on scalable, high-performance, fault-tolerant protocols for transferring large-volume data over a high-bandwidth, high-delay wide area network (WAN). The commonly used protocol for WAN data distribution is the GridFTP protocol. GridFTP uses encrypted sessions for data transfers and does not exchange any information with the network-layer resulting in reduced flexibility for network management at the site-level. We propose an application-aware software-defined networking (SDN) approach for providing differentiated network services for high-energy physics projects such as Compact Muon Solenoid (CMS) and Laser Interferometer Gravitational-Wave Observatory (LIGO). We demonstrate a policy-driven approach for differentiating network traffic by exploiting application- and network-layer collaboration to achieve accurate accounting of resources used by each project. We implement two strategies, a 7-3 queuing system, and a 10-3 queuing system, and show that the 10-3 strategy provides an additional capacity improvement of 11.74% over the 7-3 strategy.},
keywords = {application-aware SDN, application-aware software-defined networking, application-awareness, Compact Muon Solenoid, Cryptography, Data transfer, data-intensive science, data-intensive science projects, differentiated network services, DiffServ networks, Engines, fault-tolerant protocols, gravitational wave detectors, gridftp, GridFTP protocol, high-delay wide area network, high-energy physics projects, Laser Interferometer Gravitational-Wave Observatory, Metadata, physics computing, policy-driven approach, Protocols, queueing theory, queuing system, Servers, software defined networking, software defined networks, Wide area networks},
pubstate = {published},
tppubtype = {inproceedings}
}
Data-intensive science projects rely on scalable, high-performance, fault-tolerant protocols for transferring large-volume data over a high-bandwidth, high-delay wide area network (WAN). The commonly used protocol for WAN data distribution is the GridFTP protocol. GridFTP uses encrypted sessions for data transfers and does not exchange any information with the network-layer resulting in reduced flexibility for network management at the site-level. We propose an application-aware software-defined networking (SDN) approach for providing differentiated network services for high-energy physics projects such as Compact Muon Solenoid (CMS) and Laser Interferometer Gravitational-Wave Observatory (LIGO). We demonstrate a policy-driven approach for differentiating network traffic by exploiting application- and network-layer collaboration to achieve accurate accounting of resources used by each project. We implement two strategies, a 7-3 queuing system, and a 10-3 queuing system, and show that the 10-3 strategy provides an additional capacity improvement of 11.74% over the 7-3 strategy.
3.
Nadig, Deepak; Ramamurthy, Byrav
ScienceSDS: A Novel Software Defined Security Framework for Large-scale Data-intensive Science Proceedings Article
In: Proceedings of the ACM International Workshop on Security in Software Defined Networks & Network Function Virtualization, pp. 13–18, ACM, New York, NY, USA, 2017, ISBN: 978-1-4503-4908-6, (event-place: Scottsdale, Arizona, USA).
Abstract | BibTeX | Tags: data-intensive science, service function chaining, software defined security | Links:
@inproceedings{nadig_sciencesds:_2017,
title = {ScienceSDS: A Novel Software Defined Security Framework for Large-scale Data-intensive Science},
author = {Deepak Nadig and Byrav Ramamurthy},
url = {http://doi.acm.org/10.1145/3040992.3040999},
doi = {10.1145/3040992.3040999},
isbn = {978-1-4503-4908-6},
year = {2017},
date = {2017-01-01},
urldate = {2019-02-07},
booktitle = {Proceedings of the ACM International Workshop on Security in Software Defined Networks & Network Function Virtualization},
pages = {13--18},
publisher = {ACM},
address = {New York, NY, USA},
series = {SDN-NFVSec '17},
abstract = {Experimental science workflows from projects such as Compact Muon Solenoid (CMS) and Laser Interferometer Gravitational Wave Observatory (LIGO) are characterized by data-intensive computational tasks over large datasets transferred over encrypted channels. The Science DMZ approach to network design favors lossless packet forwarding through a separate isolated network over secure lossy forwarding through stateful packet processors (e.g. firewalls). We propose ScienceSDS, a novel software defined security framework for securely monitoring large-scale science datasets over a software defined networking and network functions virtualization (SDN/NFV) infrastructure.},
note = {event-place: Scottsdale, Arizona, USA},
keywords = {data-intensive science, service function chaining, software defined security},
pubstate = {published},
tppubtype = {inproceedings}
}
Experimental science workflows from projects such as Compact Muon Solenoid (CMS) and Laser Interferometer Gravitational Wave Observatory (LIGO) are characterized by data-intensive computational tasks over large datasets transferred over encrypted channels. The Science DMZ approach to network design favors lossless packet forwarding through a separate isolated network over secure lossy forwarding through stateful packet processors (e.g. firewalls). We propose ScienceSDS, a novel software defined security framework for securely monitoring large-scale science datasets over a software defined networking and network functions virtualization (SDN/NFV) infrastructure.
2016
2.
Nadig, Deepak; Zhang, Zhe; Ramamurthy, Byrav; Bockelman, Brian; Attebury, Garhan; Swanson, David
SNAG: SDN-managed Network Architecture for GridFTP Transfers Proceedings Article
In: Proceedings of the Third Workshop on Innovating the Network for Data-Intensive Science, INDIS, 2016.
BibTeX | Tags: | Links:
@inproceedings{nadig_snag:_2016,
title = {SNAG: SDN-managed Network Architecture for GridFTP Transfers},
author = {Deepak Nadig and Zhe Zhang and Byrav Ramamurthy and Brian Bockelman and Garhan Attebury and David Swanson},
url = {https://deepaknadig.com/wp-content/uploads/2021/09/Anantha-et-al.-2016-SNAG-SDN-managed-Network-Architecture-for-GridFTP.pdf},
year = {2016},
date = {2016-01-01},
urldate = {2016-01-01},
booktitle = {Proceedings of the Third Workshop on Innovating the Network for Data-Intensive Science, INDIS},
volume = {16},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2008
1.
Nadig, Deepak; Thareja, R.; Nikhil, N. A.
Performance analysis and evaluation of delay-tolerant network bundling protocol on a scalable virtual network test platform Proceedings Article
In: 2008 IET International Conference on Wireless, Mobile and Multimedia Networks, pp. 52–55, 2008.
Abstract | BibTeX | Tags: delay-tolerant network bundling protocol, delays, Internet, Internet protocol, scalable virtual network, TCP/UDP, transport protocols | Links:
@inproceedings{nadig_performance_2008,
title = {Performance analysis and evaluation of delay-tolerant network bundling protocol on a scalable virtual network test platform},
author = {Deepak Nadig and R. Thareja and N. A. Nikhil},
doi = {10.1049/cp:20080143},
year = {2008},
date = {2008-01-01},
booktitle = {2008 IET International Conference on Wireless, Mobile and Multimedia Networks},
pages = {52--55},
abstract = {The use of Internet protocol suite of TCP/UDP in environments characterized by high delay and high link error rates result in significant degradation of the Protocol performance. The DTN bundle protocol can be used in such scenarios. Performance evaluation of delay-tolerant network bundle protocol in a live network is difficult due to the absence of networks characterized by extreme environs. Control on specific performance metrics like link delay, bandwidth, connectivity, traffic flow and queue sizes are thus rendered impossible without the incorporation of a control system either in software implementation or network hardware. The control over specific environments in real- world deployments and the analysis of the protocol deployment in the above provides an understanding into the performance of the bundling protocol in harsh networking environments. This paper presents the analysis and evaluation of performance of a delay-tolerant network in a virtual test platform setup.},
keywords = {delay-tolerant network bundling protocol, delays, Internet, Internet protocol, scalable virtual network, TCP/UDP, transport protocols},
pubstate = {published},
tppubtype = {inproceedings}
}
The use of Internet protocol suite of TCP/UDP in environments characterized by high delay and high link error rates result in significant degradation of the Protocol performance. The DTN bundle protocol can be used in such scenarios. Performance evaluation of delay-tolerant network bundle protocol in a live network is difficult due to the absence of networks characterized by extreme environs. Control on specific performance metrics like link delay, bandwidth, connectivity, traffic flow and queue sizes are thus rendered impossible without the incorporation of a control system either in software implementation or network hardware. The control over specific environments in real- world deployments and the analysis of the protocol deployment in the above provides an understanding into the performance of the bundling protocol in harsh networking environments. This paper presents the analysis and evaluation of performance of a delay-tolerant network in a virtual test platform setup.