At Cisco Live 2023, Cisco announced Cisco Secure Access; a cloud-delivered Security Service Edge (SSE) product that provides comprehensive security capabilities converged in one solution, providing … Read more on Cisco Blogs

​

At Cisco Live 2023, Cisco announced Cisco Secure Access; a cloud-delivered Security Service Edge (SSE) product that provides comprehensive security capabilities converged in one solution, providing Zero Trust Network Access (ZTNA), Secure Web Gateway (SWG), Cloud Access Security Broker (CASB), Firewall-as-a-Service (FWaaS), DNS security and filtering, and Remote Access VPN capability in a unified management experience with centralized policy creation and aggregated reporting capabilities.

Traditionally, these security functions were supported by multiple point solutions; with products and their respective functions stitched together to address evolving IT requirements and secure the network. While this approach can and has worked, it significantly increases the complexity, and results in an environment that is difficult to monitor and manage effectively due to multiple administrative interfaces, less than optimal integrations, and gaps in security due to multiple point product solutions and vendors. This often results in high complexity, less security efficacy, high operational costs, and a poor end-user experience.

To address this problem, Cisco Secure Access was designed and built with three main guiding principles:

Better for users – Deliver a universal experience that seamlessly and securely connects any user to any app over any port or protocol.
Easier for IT – Simplify deployment and operations with a single console, unified client, and centralized policy management.
Safer for everyone – Mitigate risk with advanced security to maintain business continuity and avoid the repercussions of a security breach.

Comprehensive integration is key to effective security

A typical enterprise has roughly 76 security related tools in their arsenal to address various security needs, and a multi-vendor patchwork approach to solving threats places the burden of security tools integration on the end-customer, exacerbating the problem of operational complexity and resulting in increased costs. Previous generation SSE products also had many issues such as a lack of support for certain types of applications, complex product packaging, and requiring costly add-on features to get desired functionality. With SD-WAN now evolving towards SASE (or SD-WAN with SSE), the transition from multi-vendor solutions towards single-vendor solutions is also accelerating due to the need to simplify the environment and enable better security efficacy and efficiency. In fact, according to a recent Gartner survey, 75% of organizations are now seeking to consolidate security solutions and reduce the number of vendors in order to improve their risk posture.

Cisco Secure Access is the logical next-generation SSE platform of choice

Cisco Secure Access differentiates from the competition; it provides the most flexible ZTNA offering on the market, combining VPN-as-a-Service (VPNaaS) with client-based and clientless ZTNA, which makes it capable of supporting ANY application over ANY port or protocol, including Internet based, SaaS, and private applications. In short, “users simply login and get to work” in the most efficient and secure way available.

Unlike traditional ZTNA that is built with a reverse proxy architecture, Cisco takes a unique approach through a more modern Zero Trust access relay architecture. This reduces the attack surface and enables an enhanced level of enterprise privacy by giving organizations more control over their data and inspection points. It enables them to easily create policies that enforce whether specific traffic is routed through cloud security or directly to their edge security device.

Secure Access supports the key network use cases and provides unified security functions while with a unified management dashboard with a new and intuitive admin interface designed with simplicity, efficiency, and efficacy in mind. Secure Access is a SSE product that provides comprehensive, best-of-breed security capabilities such as:

Secure Web Gateway (SWG) – providing proxy web traffic, URL filtering, content filtering, and advanced application controls.
Cloud Access Security Broker (CASB) – provides cloud app discovery, risk scoring, blocking, cloud malware detection, and tenant controls.
Data Loss Prevention (DLP) – provides the ability to define and quarantine files that violate DLP rules, preventing leakage of sensitive information for supported applications.
Firewall-as-a-Service (FWaaS) – provides Layer 3/L4/L7 firewall functionality with IPS using Snort 3 technology.
DNS-layer security – prevents or limits visits to nefarious web sites, or by blocking access to designated website categories.
Remote Access – provide VPN and/or client based ZTNA for managed endpoints, or clientless ZTNA access for unmanaged endpoints with optional device posture verification (e.g., geolocation, browser type, and/or Operating System type/versions).
Remote Browser Isolation (RBI) – protects users and organizations from browser-based threats.
Secure Malware Analytics and Sandboxing capabilities – advanced sandboxing with threat intelligence into one unified solution to protect organizations from malware.
Digital Experience Monitoring – integration with ThousandEyes, enabling unparalleled visibility and ability to translate insights into actions to help resolve issues quickly and assure digital experiences across any network.

Cisco Secure Access integrates with Cisco Catalyst SD-WAN products and provides comprehensive visibility, policy controls, and reporting capabilities; one dashboard to see traffic, set policies, and analyze risk. Built on the Cisco Security Cloud, Secure Access combines all core capabilities to create a frictionless, end-user experience. It is supported by research, expertise, and intelligence from Cisco Talos; the world’s largest commercial security and threat intelligence entity, where teams of data scientists and security researchers are able to take advantage of Artificial Intelligence (AI)/Machine Learning (ML) technologies along with extensive security intelligence to enable improved security efficacy with faster detection, stronger threat correlation, deeper visibility and insights, and reduced exposure.

Massive partner opportunity as customers are transitioning towards Managed Services and business outcome based offerings

If you are a Cisco partner and offer network services to your end customers, now is the time to consider adding or supplementing security services on top of your network services offering for the following reasons:

According to a Gartner report, the Secure Access Secure Edge (SASE) and SSE market is massive with a ~36% CAGR and a world-wide opportunity approaching ~$10 BILLION by 2025!
Approximately 65% of enterprise customers are looking to adopt SSE in the next two years in order to safeguard their infrastructure/environments and improve their risk posture.
According to a 2023 Cisco Cybersecurity Readiness Index, approximately 85% of customers felt that they were not adequately prepared to handle cybersecurity threats due to distributed/complex environments making securing network connectivity difficult.

The next chapter in managed security services is here

For Cisco Partners—particularly Managed Security Service Providers (MSSPs)—there is a huge opportunity to help customers achieve a frictionless end-user experience, simplify their IT operations, and lower their security risk. Customers are now shifting towards leveraging MSSPs who can provide Network and Security as-a-Service support to address their evolving business requirements.

Per the 2023 Cisco Cybersecurity Readiness Index:

More than half (55%) of companies globally fall into the Beginner (8%) or Formative (47%) stages – meaning they are performing below average on cybersecurity readiness.
Approximately 82% of respondents said they expect a cybersecurity incident to disrupt their business in the next 12 to 24 months.
Approximately 86% of respondents said their organizations plan to increase their cybersecurity budget by at least 10% over the next 12 months.

To that end, customers are seeking the guidance of MSSPs and are looking for business outcomes such as:

Comprehensive 24x7x365 monitoring and accountability.  MSSPs that can provide around the clock monitoring of the infrastructure and be responsible for proactively monitoring the environment, managing, and respond accordingly.
MSSPs further differentiate by offering tangible and additional benefits to customers, such as faster time to identify and resolve problems, increased operational efficacy, application of industry best practices, all while improving overall end user experience and delivering the desired business outcomes with agreed to service levels.  This results in lower operating costs, better efficiencies of scale, reduced business risk, which ultimately translates to freeing up precious time/resources so that customers can focus on their core business functions and not managing their IT.
Providing expertise and an even more premium experience by combining SASE/SSE with XDR, resulting in further enhanced security efficacy, faster and more accurate threat detection and response, and stronger security posture.

Cisco Partner-Enabled Managed Services offering and benefits

If you are a Cisco partner and would like to find out how to monetize this opportunity, please register for the upcoming session of the Managed Services Voice of the Engineer. In this session, our team from the Americas Partner Organization (APO) will provide an overview of Cisco Secure Access and how it can be offered as part of a Partner-Enabled Managed Services offering and explore the benefits that can be attained for both Cisco partners and your customers. If you are interested in becoming a Cisco Partner, you can learn more by visiting Cisco’s Partner Program and Cisco Partner-Enabled Managed Services.

Register for the Managed Services Voice of the Engineer session now

Learn more about everything that Cisco Secure Access has to offer

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with #CiscoPartners on social!

Cisco Partners Facebook    Cisco Partners LinkedIn

  Cisco Secure Access differentiates from the competition; it provides the most flexible ZTNA offering on the market, combining VPN-as-a-Service (VPNaaS) with client-based and clientless ZTNA, which makes it capable of supporting ANY application over ANY port or protocol, including Internet based, SaaS, and private applications. In short, “users simply login and get to work” in the most efficient and secure way available.  Read More Cisco Blogs 

In our March 2023 blog, “What is EPSS and Why Does It Matter?”, Michael Roytman, Distinguished Engineer at Cisco (former Chief Data Scientist at Kenna Security) and co-creator of EPSS, covers the role the Exploit Prediction Scoring System (EPSS) plays in a security program. To sum it up, EPSS enables practitioners to have a defensible way to forecast how likely a newly published vulnerability is to become exploited before attackers have a chance to build new ransomware or exploits.

In this blog, we’ll cover more details about EPSS, how it compares to CVSS, as well as the role it plays in Cisco Vulnerability Management’s risk scoring.

Digging Deeper: The Importance of EPSS

EPSS is an open-source, “data-driven effort for estimating the likelihood (probability) that a software vulnerability will be exploited in the wild” (FIRST.org). Its overall goal is to help security teams better prioritize vulnerability remediation work.

Fun fact: Cisco (formerly Kenna Security) licenses the patent “Exploit Prediction Based on Machine Learning” to FIRST.org to enable EPSS development.

Anonymized data from the Cisco Vulnerability Management platform was used by the creators of EPSS to compare which vulnerabilities were being exploited in the wild to which vulnerabilities organizations were remediating. The findings revealed that remediation strategies were inconsistent and ad-hoc. Based on the evidence collected that showed what was being exploited, the creators built a data model to predict exploitability.

EPSS vs CVSS: What’s the Difference?

EPSS was initially inspired by the Common Vulnerability Scoring System (CVSS). CVSS assigns scores to vulnerabilities based on their principal characteristics; the score indicates the severity of a vulnerability, providing a range from 0.0 to 10.0 (the higher the score, the greater severity). CVSS can be categorized into low, medium, and high severity, and organizations can use CVSS to help prioritize vulnerabilities that exist in the system. However, CVSS on its own doesn’t indicate a likelihood of exploitation, leading to criticisms that call out its ineffectiveness in prioritizing and predicting threats.

EPSS, on the other hand, estimates the probability that a vulnerability will be exploited in the wild in the next 30 days, with a score ranging between 0 to 1. EPSS looks at two key prioritization strategies: coverage and efficiency. Coverage is the proportion of vulnerabilities with known exploitation activity that are prioritized. Efficiency is the proportion of all prioritized vulnerabilities with known exploitation activity. Despite its ability to help in predicting which vulnerabilities will be exploited in the wild, EPSS doesn’t provide all the information needed to deprioritize vulnerabilities, which makes it difficult to make decisions on what to fix first.

Coupling EPSS and CVSS scoring data enables organizations to more effectively prioritize vulnerabilities based on both severity and probability of exploitation. Even so, there are other data sources like real-time threat data that should be incorporated into vulnerability prioritization scoring for optimized results. More on that in just a bit.

What It Means for Cisco Vulnerability Management Customers

Risk Scoring in the Cisco Vulnerability Management platform helps customers prioritize the vulnerabilities that pose the greatest risk to their specific organizations, while deprioritizing the ones that don’t. Our risk score is continuously evolving to include the latest inputs for the most accurate prioritization. This update easily enables customers to identify and remediate top priority vulnerabilities based on the prediction that it will become an Active Internet Breach in the near future.

Figure 1: Explore page in Cisco Vulnerability Management platform

While it’s important to understand a vulnerability may be exploited in the future, it’s even more important to know which vulnerabilities are already being exploited. That’s why, in conjunction with EPSS and CVSS, Cisco Vulnerability Management risk scoring incorporates an organization’s internal security data and threat and exploit intelligence from 19+ feeds, including Cisco Talos, to not only determine how risky a vulnerability is, but to also understand the volume and velocity at which the vulnerability is being targeted. By leveraging the risk score in Cisco Vulnerability Management, customers can determine which vulnerabilities pose the biggest risk to their organization and which vulnerabilities are low risk and, therefore, can be deprioritized. The result is that customers are focusing their limited resources on remediating the vulnerabilities that matter most.

In addition to identifying which vulnerabilities are most likely to result in an exploit, Cisco Vulnerability Management uses Risk Meter scoring to also highlight the impact of those exploits by measuring the risks of assets, groups of assets, and organizations. With accurate and quantifiable risk scores, customers can understand their organizations’ current risk posture and identify the actions needed to reduce the greatest amount of risk.

Interested in learning more about EPSS? Check out the site and browse the data (it’s open and free): www.first.org/epss

Want to take a deeper look at Cisco Vulnerability Management? Visit our page: https://www.cisco.com/site/us/en/products/security/vulnerability-management/index.html

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

Instagram
Facebook
Twitter
LinkedIn

​

In our March 2023 blog, “What is EPSS and Why Does It Matter?”, Michael Roytman, Distinguished Engineer at Cisco (former Chief Data Scientist at Kenna Security) and co-creator of EPSS, covers the role the Exploit Prediction Scoring System (EPSS) plays in a security program. To sum it up, EPSS enables practitioners to have a defensible way to forecast how likely a newly published vulnerability is to become exploited before attackers have a chance to build new ransomware or exploits.

In this blog, we’ll cover more details about EPSS, how it compares to CVSS, as well as the role it plays in Cisco Vulnerability Management’s risk scoring.

Digging Deeper: The Importance of EPSS

EPSS is an open-source, “data-driven effort for estimating the likelihood (probability) that a software vulnerability will be exploited in the wild” (FIRST.org). Its overall goal is to help security teams better prioritize vulnerability remediation work.

Fun fact: Cisco (formerly Kenna Security) licenses the patent “Exploit Prediction Based on Machine Learning” to FIRST.org to enable EPSS development.

Anonymized data from the Cisco Vulnerability Management platform was used by the creators of EPSS to compare which vulnerabilities were being exploited in the wild to which vulnerabilities organizations were remediating. The findings revealed that remediation strategies were inconsistent and ad-hoc. Based on the evidence collected that showed what was being exploited, the creators built a data model to predict exploitability.

EPSS vs CVSS: What’s the Difference?

EPSS was initially inspired by the Common Vulnerability Scoring System (CVSS). CVSS assigns scores to vulnerabilities based on their principal characteristics; the score indicates the severity of a vulnerability, providing a range from 0.0 to 10.0 (the higher the score, the greater severity). CVSS can be categorized into low, medium, and high severity, and organizations can use CVSS to help prioritize vulnerabilities that exist in the system. However, CVSS on its own doesn’t indicate a likelihood of exploitation, leading to criticisms that call out its ineffectiveness in prioritizing and predicting threats.

EPSS, on the other hand, estimates the probability that a vulnerability will be exploited in the wild in the next 30 days, with a score ranging between 0 to 1. EPSS looks at two key prioritization strategies: coverage and efficiency. Coverage is the proportion of vulnerabilities with known exploitation activity that are prioritized. Efficiency is the proportion of all prioritized vulnerabilities with known exploitation activity. Despite its ability to help in predicting which vulnerabilities will be exploited in the wild, EPSS doesn’t provide all the information needed to deprioritize vulnerabilities, which makes it difficult to make decisions on what to fix first.

Coupling EPSS and CVSS scoring data enables organizations to more effectively prioritize vulnerabilities based on both severity and probability of exploitation. Even so, there are other data sources like real-time threat data that should be incorporated into vulnerability prioritization scoring for optimized results. More on that in just a bit.

What It Means for Cisco Vulnerability Management Customers

Risk Scoring in the Cisco Vulnerability Management platform helps customers prioritize the vulnerabilities that pose the greatest risk to their specific organizations, while deprioritizing the ones that don’t. Our risk score is continuously evolving to include the latest inputs for the most accurate prioritization. This update easily enables customers to identify and remediate top priority vulnerabilities based on the prediction that it will become an Active Internet Breach in the near future.

Figure 1: Explore page in Cisco Vulnerability Management platform

While it’s important to understand a vulnerability may be exploited in the future, it’s even more important to know which vulnerabilities are already being exploited. That’s why, in conjunction with EPSS and CVSS, Cisco Vulnerability Management risk scoring incorporates an organization’s internal security data and threat and exploit intelligence from 19+ feeds, including Cisco Talos, to not only determine how risky a vulnerability is, but to also understand the volume and velocity at which the vulnerability is being targeted. By leveraging the risk score in Cisco Vulnerability Management, customers can determine which vulnerabilities pose the biggest risk to their organization and which vulnerabilities are low risk and, therefore, can be deprioritized. The result is that customers are focusing their limited resources on remediating the vulnerabilities that matter most.

In addition to identifying which vulnerabilities are most likely to result in an exploit, Cisco Vulnerability Management uses Risk Meter scoring to also highlight the impact of those exploits by measuring the risks of assets, groups of assets, and organizations. With accurate and quantifiable risk scores, customers can understand their organizations’ current risk posture and identify the actions needed to reduce the greatest amount of risk.

Interested in learning more about EPSS? Check out the site and browse the data (it’s open and free): www.first.org/epss

Want to take a deeper look at Cisco Vulnerability Management? Visit our page: https://www.cisco.com/site/us/en/products/security/vulnerability-management/index.html

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

InstagramFacebookTwitterLinkedIn

  Learn about EPSS, how it compares to CVSS, and the role it plays in Cisco Vulnerability Management’s risk scoring.  Read More Cisco Blogs 

Cisco is aware of reports that Akira ransomware threat actors have been targeting Cisco VPNs that are not configured for multi-factor authentication to infiltrate organizations, and we have observed instances where threat actors appear to be targeting organizations that do not configure multi-factor authentication for their VPN users.

This highlights the importance of enabling multi-factor authentication (MFA) in VPN implementations. By implementing MFA, organizations can significantly reduce the risk of unauthorized access, including a potential ransomware infection. If a threat actor successfully gains unauthorized access to a user’s VPN credentials, such as through brute force attacks, MFA provides an additional layer of protection to prevent the threat actors from gaining access to the VPN.

Cisco has been actively collaborating with Rapid7 in the investigation of similar attack tactics. Cisco would like to thank Rapid7 for their valuable collaboration.

Akira Ransomware

Initial reports of the Akira ransomware date back to March 2023. The threat actors responsible for the Akira ransomware use different extortion strategies and operate a website on the TOR network (with a .onion domain) where they list victims and any pilfered information if the ransom demands are not met. Victims are directed to contact the attackers through this TOR-based site, using a unique identifier found in the ransom message they receive, to initiate negotiations.

Targeting VPN Implementations without MFA

When targeting VPNs in general, the first stage of the attack is carried out by taking advantage of exposed services or applications. The attackers often focus on the absence of or known vulnerabilities  in multi-factor authentication (MFA) and known vulnerabilities in VPN software. Once the attackers have obtained a foothold into a target network, they try to extract credentials through LSASS (Local Security Authority Subsystem Service) dumps to facilitate further movement within the network and elevate privileges if needed. The group has also been linked to using other tools commonly referred to as Living-Off-The-Land Binaries (LOLBins) or Commercial Off-The-Shelf (COTS) tools, such as PCHunter64, or engaging in the creation of minidumps to gather further intelligence about or pivot inside the target network.

Brute-Forcing vs. Purchasing Credentials

There are two primary ways regarding how the attackers might have gained access:

Brute-Forcing: We have seen evidence of brute force and password spraying attempts. This involves using automated tools to try many different combinations of usernames and passwords until the correct credentials are found. Password spraying is a type of brute-force attack in which an attacker attempts to gain unauthorized access to a large number of accounts by trying a few common passwords against many usernames. Unlike traditional brute-force attacks, where every possible password is tried for one user, password spraying focuses on trying a few passwords across many accounts, often avoiding account lockouts and detection. If the VPN configurations had more robust logging, it might be possible to see evidence of a brute-force attack, such as multiple failed login attempts. The following logs from a Cisco ASA can allow you to detect potential brute force attacks:

Login attempts with invalid username/password (%ASA-6-113015)
Example:
%ASA-6-113015: AAA user authentication Rejected: reason = reason : local database: user = user: user IP = xxx.xxx.xxx.xxx
Remote access VPN session creation attempts for unexpected connection profiles/tunnel groups (%ASA-4-113019, %ASA-4-722041, or %ASA-7-734003)

Purchasing Credentials through Dark Web Market: Attackers can sometimes acquire valid credentials by purchasing them on the dark web, an encrypted part of the internet often associated with illegal activities. These credentials might be available due to previous data breaches or through other means. Acquiring credentials in this way would likely leave no trace in the VPN’s logs, as the attacker would simply log in using valid credentials.

Logging within Cisco’s ASA

Logging is a crucial part of cybersecurity that involves recording events happening within a system. In the reported attack scenarios, the logging was not configured in the affected Cisco’s ASAs. This has made it challenging to determine precisely how the Akira ransomware attackers were able to access the VPNs. The absence of detailed logs leaves gaps in understanding, hindering a clear analysis of the attack method.

To set up logging on a Cisco ASA you can easily access the command-line interface (CLI) and use the logging enable, logging host, and logging trap commands to specify the logging server, severity levels, and other parameters. Sending logging data to a remote syslog server is recommended. This enables improved correlation and auditing of network and security incidents across various network devices.

Refer to the Guide to Secure the Cisco ASA Firewall to get detailed information about best practices to configure logging and secure a Cisco ASA.

Additional Forensics Guidance for Incident Responders

Refer to the Cisco ASA Forensics Guide for First Responders to obtain instructions on how to collect evidence from Cisco ASA devices. The document lists different commands that can be executed to assemble evidence for a probe, along with the corresponding output that needs to be captured when these commands are run. In addition, the document explains how to conduct integrity checks on the system images of Cisco ASA devices and details a method for gathering a core file or memory dump from such a device.

Cisco will remain vigilant in monitoring and investigating these activities and will update customers with any new findings or information.

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

Instagram
Facebook
Twitter
LinkedIn

​

Cisco is aware of reports that Akira ransomware threat actors have been targeting Cisco VPNs that are not configured for multi-factor authentication to infiltrate organizations, and we have observed instances where threat actors appear to be targeting organizations that do not configure multi-factor authentication for their VPN users.

This highlights the importance of enabling multi-factor authentication (MFA) in VPN implementations. By implementing MFA, organizations can significantly reduce the risk of unauthorized access, including a potential ransomware infection. If a threat actor successfully gains unauthorized access to a user’s VPN credentials, such as through brute force attacks, MFA provides an additional layer of protection to prevent the threat actors from gaining access to the VPN.

Cisco has been actively collaborating with Rapid7 in the investigation of similar attack tactics. Cisco would like to thank Rapid7 for their valuable collaboration.

Akira Ransomware

Initial reports of the Akira ransomware date back to March 2023. The threat actors responsible for the Akira ransomware use different extortion strategies and operate a website on the TOR network (with a .onion domain) where they list victims and any pilfered information if the ransom demands are not met. Victims are directed to contact the attackers through this TOR-based site, using a unique identifier found in the ransom message they receive, to initiate negotiations.

Targeting VPN Implementations without MFA

When targeting VPNs in general, the first stage of the attack is carried out by taking advantage of exposed services or applications. The attackers often focus on the absence of or known vulnerabilities  in multi-factor authentication (MFA) and known vulnerabilities in VPN software. Once the attackers have obtained a foothold into a target network, they try to extract credentials through LSASS (Local Security Authority Subsystem Service) dumps to facilitate further movement within the network and elevate privileges if needed. The group has also been linked to using other tools commonly referred to as Living-Off-The-Land Binaries (LOLBins) or Commercial Off-The-Shelf (COTS) tools, such as PCHunter64, or engaging in the creation of minidumps to gather further intelligence about or pivot inside the target network.

Brute-Forcing vs. Purchasing Credentials

There are two primary ways regarding how the attackers might have gained access:

Brute-Forcing: We have seen evidence of brute force and password spraying attempts. This involves using automated tools to try many different combinations of usernames and passwords until the correct credentials are found. Password spraying is a type of brute-force attack in which an attacker attempts to gain unauthorized access to a large number of accounts by trying a few common passwords against many usernames. Unlike traditional brute-force attacks, where every possible password is tried for one user, password spraying focuses on trying a few passwords across many accounts, often avoiding account lockouts and detection. If the VPN configurations had more robust logging, it might be possible to see evidence of a brute-force attack, such as multiple failed login attempts. The following logs from a Cisco ASA can allow you to detect potential brute force attacks:
Login attempts with invalid username/password (%ASA-6-113015)
Example:
%ASA-6-113015: AAA user authentication Rejected: reason = reason : local database: user = user: user IP = xxx.xxx.xxx.xxx
Remote access VPN session creation attempts for unexpected connection profiles/tunnel groups (%ASA-4-113019, %ASA-4-722041, or %ASA-7-734003)
Purchasing Credentials through Dark Web Market: Attackers can sometimes acquire valid credentials by purchasing them on the dark web, an encrypted part of the internet often associated with illegal activities. These credentials might be available due to previous data breaches or through other means. Acquiring credentials in this way would likely leave no trace in the VPN’s logs, as the attacker would simply log in using valid credentials.

Logging within Cisco’s ASA

Logging is a crucial part of cybersecurity that involves recording events happening within a system. In the reported attack scenarios, the logging was not configured in the affected Cisco’s ASAs. This has made it challenging to determine precisely how the Akira ransomware attackers were able to access the VPNs. The absence of detailed logs leaves gaps in understanding, hindering a clear analysis of the attack method.

To set up logging on a Cisco ASA you can easily access the command-line interface (CLI) and use the logging enable, logging host, and logging trap commands to specify the logging server, severity levels, and other parameters. Sending logging data to a remote syslog server is recommended. This enables improved correlation and auditing of network and security incidents across various network devices.

Refer to the Guide to Secure the Cisco ASA Firewall to get detailed information about best practices to configure logging and secure a Cisco ASA.

Additional Forensics Guidance for Incident Responders

Refer to the Cisco ASA Forensics Guide for First Responders to obtain instructions on how to collect evidence from Cisco ASA devices. The document lists different commands that can be executed to assemble evidence for a probe, along with the corresponding output that needs to be captured when these commands are run. In addition, the document explains how to conduct integrity checks on the system images of Cisco ASA devices and details a method for gathering a core file or memory dump from such a device.

Cisco will remain vigilant in monitoring and investigating these activities and will update customers with any new findings or information.

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

InstagramFacebookTwitterLinkedIn

  Cisco has observed instances where threat actors appear to be targeting organizations that do not configure multi-factor authentication
for their VPN users. This highlights the importance of enabling multi-factor authentication (MFA) in VPN implementations.  Read More Cisco Blogs 

After the executive order to bolster the nations cybersecurity following the Colonial Pipeline attack, the U.S. Transportation Security Administration (TSA) has been releasing new mandates for critical infrastructure such as freight and passenger rail, pipelines, and airports, with more industries to follow.

The networks that support these critical infrastructures are mission-critical, which means that it is essential to be able to stay connected while securely administering policy in the industrial space. Being an industry leader in networking and security across both the information technology (IT) and operational technology (OT) domains, Cisco is in a unique position to deliver an end-to-end security strategy, while enhancing operational uptime and resiliency.

To strengthen the cybersecurity posture of the nation’s critical infrastructure, there are four key requirements outlined by the mandates, highlighted in bold text below.

Network segmentation

The first requirement is to “Implement network segmentation policies and controls to ensure that the Operational Technology (OT) system can continue to safely operate if an Information Technology (IT) system has been compromised.”

Using a defense-in-depth approach, Cisco addresses this requirement in many parts of the network, adapting to the unique architecture needs of individual organizations. The solution is a common one, use the network infrastructure to segment a network. Do not wait until you reach a “security appliance” to do security. Cisco provides an end-to-end segmentation solution in which data is kept within its own virtual network from source to destination, wherever that may be.

To support the unique requirements of industrial networks, the reach of Cisco SD-WAN has been expanded through Cisco Industrial Routers, which provide the connectivity, mobility, and security required for critical infrastructure. SD-WAN segments traffic at the edge of the network and maintains separation through all relevant points in the network. Policy can be orchestrated across multiple enforcement points in the network using Cisco Catalyst SD-WAN, or—if your organization prefers—can support the evolution to a secure service edge (SSE) with Cisco Secure Access.

Access control

TSA highlights the need to “Implement access control measures to secure and prevent unauthorized access to Critical Cyber Systems.” As OT devices traverse both the LAN and the WAN with a unified identity, Cisco can enforce policy everywhere. Cisco Security Group Tags (SGTs) identify the role that a device has on the network, and the associated privileges are enforced by switches, routers, and firewalls, depending on where the data flows.

Remote users, whether internal technicians or vendor support, often need access to critical cyber systems. Cisco Secure Equipment Access (SEA) provides flexible access for remote configuration and maintenance of industrial assets in distributed locations while minimizing security risk.

Continuous monitoring

Segmentation is not enough to complete a security solution. By implementing “continuous monitoring and detection policies and procedures to detect cybersecurity threats and correct anomalies that affect Critical Cyber System operations,” we can continually monitor and evaluate the trust of both users and devices on our networks and push policy back into the network as security posture changes.

To provide visibility and security posture to the industrial network, Cisco Cyber Vision is embedded in Cisco networking infrastructure in order to avoid the need for dedicated appliances and/or costly Switched Port Analyzer (SPAN) solutions. Cyber Vision identifies assets, their characteristics, and their communication patterns to “reduce the risk of exploitation of unpatched systems through the application of security patches and updates for operating systems, applications, drivers and firmware on Critical Cyber Systems in a timely manner using a risk-based methodology.” Cyber Vision automatically identifies device vulnerabilities and calculates risk scores so you can proactively build an improvement process to address risks.

Cisco’s capabilities, highlighted above, not only meet the current TSA Cybersecurity Directive requirements but also enable clients to deliver more robust cybersecurity capabilities to thwart efforts by industry threats. Most significantly, these capabilities are foundational for enabling both security and operational resiliency as well as optimizing the performance of mission-critical networks.

 

To learn more about how Cisco can help you secure your industrial operations, please contact us or visit cisco.com/go/iotsecurity. And don’t forget to subscribe to our OT security newsletter.

​

After the executive order to bolster the nations cybersecurity following the Colonial Pipeline attack, the U.S. Transportation Security Administration (TSA) has been releasing new mandates for critical infrastructure such as freight and passenger rail, pipelines, and airports, with more industries to follow.

The networks that support these critical infrastructures are mission-critical, which means that it is essential to be able to stay connected while securely administering policy in the industrial space. Being an industry leader in networking and security across both the information technology (IT) and operational technology (OT) domains, Cisco is in a unique position to deliver an end-to-end security strategy, while enhancing operational uptime and resiliency.

To strengthen the cybersecurity posture of the nation’s critical infrastructure, there are four key requirements outlined by the mandates, highlighted in bold text below.

Network segmentation

The first requirement is to “Implement network segmentation policies and controls to ensure that the Operational Technology (OT) system can continue to safely operate if an Information Technology (IT) system has been compromised.”

Using a defense-in-depth approach, Cisco addresses this requirement in many parts of the network, adapting to the unique architecture needs of individual organizations. The solution is a common one, use the network infrastructure to segment a network. Do not wait until you reach a “security appliance” to do security. Cisco provides an end-to-end segmentation solution in which data is kept within its own virtual network from source to destination, wherever that may be.

To support the unique requirements of industrial networks, the reach of Cisco SD-WAN has been expanded through Cisco Industrial Routers, which provide the connectivity, mobility, and security required for critical infrastructure. SD-WAN segments traffic at the edge of the network and maintains separation through all relevant points in the network. Policy can be orchestrated across multiple enforcement points in the network using Cisco Catalyst SD-WAN, or—if your organization prefers—can support the evolution to a secure service edge (SSE) with Cisco Secure Access.

Access control

TSA highlights the need to “Implement access control measures to secure and prevent unauthorized access to Critical Cyber Systems.” As OT devices traverse both the LAN and the WAN with a unified identity, Cisco can enforce policy everywhere. Cisco Security Group Tags (SGTs) identify the role that a device has on the network, and the associated privileges are enforced by switches, routers, and firewalls, depending on where the data flows.

Remote users, whether internal technicians or vendor support, often need access to critical cyber systems. Cisco Secure Equipment Access (SEA) provides flexible access for remote configuration and maintenance of industrial assets in distributed locations while minimizing security risk.

Continuous monitoring

Segmentation is not enough to complete a security solution. By implementing “continuous monitoring and detection policies and procedures to detect cybersecurity threats and correct anomalies that affect Critical Cyber System operations,” we can continually monitor and evaluate the trust of both users and devices on our networks and push policy back into the network as security posture changes.

To provide visibility and security posture to the industrial network, Cisco Cyber Vision is embedded in Cisco networking infrastructure in order to avoid the need for dedicated appliances and/or costly Switched Port Analyzer (SPAN) solutions. Cyber Vision identifies assets, their characteristics, and their communication patterns to “reduce the risk of exploitation of unpatched systems through the application of security patches and updates for operating systems, applications, drivers and firmware on Critical Cyber Systems in a timely manner using a risk-based methodology.” Cyber Vision automatically identifies device vulnerabilities and calculates risk scores so you can proactively build an improvement process to address risks.

Cisco’s capabilities, highlighted above, not only meet the current TSA Cybersecurity Directive requirements but also enable clients to deliver more robust cybersecurity capabilities to thwart efforts by industry threats. Most significantly, these capabilities are foundational for enabling both security and operational resiliency as well as optimizing the performance of mission-critical networks.

To learn more about how Cisco can help you secure your industrial operations, please contact us or visit cisco.com/go/iotsecurity. And don’t forget to subscribe to our OT security newsletter.

  Protecting the nation’s critical infrastructure requires strong cybersecurity controls. Being an industry leader in networking and security across both IT and OT, Cisco is uniquely positioned to help you comply with the latest TSA mandates. Learn what the requirements are.  Read More Cisco Blogs 

In today’s fast-paced and hyper-connected world, gone are the days when deploying network devices required sending an expert to each location — a cumbersome, time-consuming, and error-prone process that caused significant downtime and increased operational costs. To surmount these barriers, Cisco offers a variety of network orchestrators. These included Cisco Catalyst Center (formerly Cisco DNA Center), SD WAN Manager (formerly Cisco vManage), and Meraki Dashboard, which assist businesses in automating their campus network management including Day 0 provisioning. These orchestrators allow network administrators to remotely deploy a large number of network devices quickly and securely, without requiring any human intervention. This not only saves time and money but also liberates IT department resources, allowing them to redirect their efforts towards other critical areas.

Using Catalyst Center PnP, Cisco IT was able to reduce annual deployment costs for some sites by approximately 25%, or more than $1.6 million. Additionally, upgrading our 285 small and medium-sized offices with Cisco Catalyst Center saved 570 man-hours per upgrade[1].

In addition to Cisco network orchestrators for customers employing a Do-It-Yourself (DIY) approach with homegrown tools, Catalyst 9000 series switches offer support for an assortment of open standard-based implementations for Day 0 network automation, such as Preboot eXecution Environment (PXE) and Zero Touch Provisioning (ZTP). So, if you find yourself still manually configuring network devices, it may be time to consider stepping out of the stone age and exploring the benefits of automation.

Day 0 network automation

When delving into the realm of open standard-based Day 0 network automation, it becomes clear that PXE, while a useful technique, comes with a set of limitations, such as only allowing network devices to boot from a network-based source and not being able to send configurations to devices during the PXE workflow. ZTP, on the other hand, can be used to upgrade software images and push configuration files, reducing the chance of human error and ensuring configuration consistency in order to get network devices up and running.

While ZTP and PXE are convenient for automating the provisioning process, they may inadvertently expose network devices to potential threats. Lack of secure authentication and verification mechanisms during the provisioning process is one of the primary concerns with these techniques. Furthermore, ZTP and PXE utilize HTTP/TFTP to download the software image or configuration files, which are inherently insecure protocols because they lack encryption. Due to these limitations, these techniques could result in unauthorized access to the device or a man-in-the-middle attack if the right security measures aren’t put in place during the device provisioning.

Cyberattacks have increased

In today’s rapidly evolving digital landscape, where enterprises are undergoing substantial transformation, cyberattacks have increased amid the rise of cloud computing, hybrid and multi-cloud networks, and the rise of remote work. According to the latest IBM Ponemon Institute 2023 Cost of Data Breach Study, the average cost of a data breach reached an all-time high in 2023 of USD 4.45 million [2]. Furthermore, according to ITIC’s 2022 Global Server Hardware Security report, 76% of firms cite Data Breaches and Human Error as the leading reason of server, OS, application, and network downtime, and the hourly cost of downtime has risen to over $300,000[3].

Given that cybercriminals are constantly devising new techniques to infiltrate networks, the traditional security approach, which assumes that everything within the network perimeter is trustworthy, is no longer sufficient. This is also true for Day 0 network automation, where it is crucial to validate the trustworthiness of the newly deployed device, bootstrap server, and configurations pushed to the device. Without implementing these security measures, our networks are vulnerable to a variety of cyberattacks, including the notorious zero-day exploits. To ensure maximal security and minimize potential risks, the Zero Trust principle of “never trust, always verify” must be implemented throughout the entire provisioning process.

Maintain security throughout the provisioning process

This is where Secure Zero Touch Provisioning comes into play. Secure ZTP, as described in RFC 8572, is an enhanced version of ZTP that emphasizes maintaining security throughout the provisioning process by reducing the likelihood of security breaches. Secure ZTP is a proactive approach that employs robust authentication, a secure boot mechanism, and encrypted communication channels to enhance the security posture of a network while Day 0 network automation is in place.

How does Secure ZTP work?

Secure ZTP employs three-step validation, including device validation, server validation, and artifact validation, to securely onboard the device. The diagram provided below illustrates the various steps involved in the device onboarding and provisioning process within a secure ZTP framework. Let’s take a closer look at each of these steps:

1. Device Validation

Before onboarding a new device on the network, it is crucial to ensure that neither the device nor its firmware has been tampered with or compromised to prevent supply chain or any other attacks, in which malicious actors attempt to introduce modified or malicious devices into the network. Based on the recent IBM report, 15% of organizations identified a supply chain compromise as the source of a data breach [2].Secure ZTP performs device authentication prior to provisioning it in order to verify the integrity and authenticity of a device and to allow only authorized devices to join the network.For device validation, Secure ZTP uses certificate-based authentication where the device sends the Trust Anchor Certificate (also known as a SUDI certificate installed in the device during the manufacturing process) to the Secure ZTP server, and the server validates it with the public certificate (provided by the manufacturer) to ensure the device’s authenticity.

2. Server Validation

Server validation is another vital part of the Secure ZTP. By confirming the server’s identity, the device can assure that it is communicating with an uncompromised, trustworthy server. This prevents unauthorized or malicious servers from intercepting or manipulating the provisioning process. After verifying the device, bootstrap server sends server certificate. The device requests bootstrapping data with the flag “signed-data-preferred” after receiving the server certificate, indicating that the device does not trust the server. In this case, keep in mind that server validation is optional in Secure ZTP. If the network administrator decides to perform server validation (which entitles server to receive bootstrapping progress report), the server will send the “redirect-data” with other bootstrapping data to the device, providing its own address and the trust anchor. The device verifies the server’s certificate and marks it as trusted server after receiving the trust anchor. Here, if the system administrator opts not to validate the server, the server will instead pass on bootstrapping data in place of the “redirect-data”. In addition, the device will continue the bootstrapping process assuming the server is untrusted.

3. Artifact Validation

Artifact validation is important to ensure that the configuration files or software images used to provision network devices are authentic and have not been tampered with. Once the server validation is complete (or skipped), the bootstrap server will send the owner certificate, ownership voucher, and onboarding information to the device as bootstrapping data. Let’s discuss them closely to gain a better understanding.

Ownership Voucher (OV): The ownership voucher artifact validates the owner certificate to verify the identity of the device’s owner. The device manufacture signs the OV and provides it to the customer based on the request. To generate the OV, the customer must provide the pinned-domain-cert and serial number of the device to the Cisco MASA server.
Owner Certificate (OC): Owner Certificate is an X.509 certificate that binds an owner identity to a public key, which a device can use to validate signature over the conveyed information artifact. The owner certificate also holds all intermediate certificates that led to the “pinned-domain-cert” certificate specified in the ownership voucher, allowing the OV to validate the OC.
Conveyed Information/Onboarding Information: Onboarding information provides data necessary for a device to bootstrap itself and establish secure connections with other systems. Onboarding information specify details about the boot image a device must be running, an initial configuration the device must commit, and scripts that the device must successfully execute. The onboarding information must be signed by the device’s owner using OC.

Zero Trust is crucial when performing Day 0 provisioning

In addition to its many features, Secure ZTP goes beyond by offering audit trails and monitoring capabilities. This includes logging all provisioning events, configuration changes, and user actions. By monitoring ZTP activities, network administrators can quickly detect any suspicious activity and take appropriate action.

As we wrap up our discussion, it becomes clear that Zero Trust is also crucial when performing Day 0 provisioning, and Secure ZTP is the best way to ensure that zero trust principles are applied while performing Day 0 provisioning using a Do-It-Yourself (DIY) approach.

With the IOS-XE 17.11.1 release, users can now take advantage of the secure Zero Touch Provisioning (ZTP) capabilities with Catalyst 9000 series switches. This exciting feature aligns with the specifications outlined in RFC 8572, ensuring a secure and seamless provisioning experience. For more details about how to implement Secure ZTP, please refer the IOS-XE 17.11.1 Configuration Guide.

Keep Learning with these resources

Visit the Zero Touch Resource Hub
Learn more about Cisco IOS XE
Cisco IOS XE blogs
Zero Touch Provisioning blog

References

Cisco DNA Center: Early Results from Intent-based Networking
Security, Data Breaches Top Cause of Downtime in 2022
IBM – Cost of a Data Breach Report 2023

​

In today’s fast-paced and hyper-connected world, gone are the days when deploying network devices required sending an expert to each location — a cumbersome, time-consuming, and error-prone process that caused significant downtime and increased operational costs. To surmount these barriers, Cisco offers a variety of network orchestrators. These included Cisco Catalyst Center (formerly Cisco DNA Center), SD WAN Manager (formerly Cisco vManage), and Meraki Dashboard, which assist businesses in automating their campus network management including Day 0 provisioning. These orchestrators allow network administrators to remotely deploy a large number of network devices quickly and securely, without requiring any human intervention. This not only saves time and money but also liberates IT department resources, allowing them to redirect their efforts towards other critical areas.

Using Catalyst Center PnP, Cisco IT was able to reduce annual deployment costs for some sites by approximately 25%, or more than $1.6 million. Additionally, upgrading our 285 small and medium-sized offices with Cisco Catalyst Center saved 570 man-hours per upgrade[1].

In addition to Cisco network orchestrators for customers employing a Do-It-Yourself (DIY) approach with homegrown tools, Catalyst 9000 series switches offer support for an assortment of open standard-based implementations for Day 0 network automation, such as Preboot eXecution Environment (PXE) and Zero Touch Provisioning (ZTP). So, if you find yourself still manually configuring network devices, it may be time to consider stepping out of the stone age and exploring the benefits of automation.

Day 0 network automation

When delving into the realm of open standard-based Day 0 network automation, it becomes clear that PXE, while a useful technique, comes with a set of limitations, such as only allowing network devices to boot from a network-based source and not being able to send configurations to devices during the PXE workflow. ZTP, on the other hand, can be used to upgrade software images and push configuration files, reducing the chance of human error and ensuring configuration consistency in order to get network devices up and running.

While ZTP and PXE are convenient for automating the provisioning process, they may inadvertently expose network devices to potential threats. Lack of secure authentication and verification mechanisms during the provisioning process is one of the primary concerns with these techniques. Furthermore, ZTP and PXE utilize HTTP/TFTP to download the software image or configuration files, which are inherently insecure protocols because they lack encryption. Due to these limitations, these techniques could result in unauthorized access to the device or a man-in-the-middle attack if the right security measures aren’t put in place during the device provisioning.

Cyberattacks have increased

In today’s rapidly evolving digital landscape, where enterprises are undergoing substantial transformation, cyberattacks have increased amid the rise of cloud computing, hybrid and multi-cloud networks, and the rise of remote work. According to the latest IBM Ponemon Institute 2023 Cost of Data Breach Study, the average cost of a data breach reached an all-time high in 2023 of USD 4.45 million [2]. Furthermore, according to ITIC’s 2022 Global Server Hardware Security report, 76% of firms cite Data Breaches and Human Error as the leading reason of server, OS, application, and network downtime, and the hourly cost of downtime has risen to over $300,000[3].

Given that cybercriminals are constantly devising new techniques to infiltrate networks, the traditional security approach, which assumes that everything within the network perimeter is trustworthy, is no longer sufficient. This is also true for Day 0 network automation, where it is crucial to validate the trustworthiness of the newly deployed device, bootstrap server, and configurations pushed to the device. Without implementing these security measures, our networks are vulnerable to a variety of cyberattacks, including the notorious zero-day exploits. To ensure maximal security and minimize potential risks, the Zero Trust principle of “never trust, always verify” must be implemented throughout the entire provisioning process.

Maintain security throughout the provisioning process

This is where Secure Zero Touch Provisioning comes into play. Secure ZTP, as described in RFC 8572, is an enhanced version of ZTP that emphasizes maintaining security throughout the provisioning process by reducing the likelihood of security breaches. Secure ZTP is a proactive approach that employs robust authentication, a secure boot mechanism, and encrypted communication channels to enhance the security posture of a network while Day 0 network automation is in place.

How does Secure ZTP work?

Secure ZTP employs three-step validation, including device validation, server validation, and artifact validation, to securely onboard the device. The diagram provided below illustrates the various steps involved in the device onboarding and provisioning process within a secure ZTP framework. Let’s take a closer look at each of these steps:

1. Device Validation

Before onboarding a new device on the network, it is crucial to ensure that neither the device nor its firmware has been tampered with or compromised to prevent supply chain or any other attacks, in which malicious actors attempt to introduce modified or malicious devices into the network. Based on the recent IBM report, 15% of organizations identified a supply chain compromise as the source of a data breach [2].Secure ZTP performs device authentication prior to provisioning it in order to verify the integrity and authenticity of a device and to allow only authorized devices to join the network.For device validation, Secure ZTP uses certificate-based authentication where the device sends the Trust Anchor Certificate (also known as a SUDI certificate installed in the device during the manufacturing process) to the Secure ZTP server, and the server validates it with the public certificate (provided by the manufacturer) to ensure the device’s authenticity.

2. Server Validation

Server validation is another vital part of the Secure ZTP. By confirming the server’s identity, the device can assure that it is communicating with an uncompromised, trustworthy server. This prevents unauthorized or malicious servers from intercepting or manipulating the provisioning process. After verifying the device, bootstrap server sends server certificate. The device requests bootstrapping data with the flag “signed-data-preferred” after receiving the server certificate, indicating that the device does not trust the server. In this case, keep in mind that server validation is optional in Secure ZTP. If the network administrator decides to perform server validation (which entitles server to receive bootstrapping progress report), the server will send the “redirect-data” with other bootstrapping data to the device, providing its own address and the trust anchor. The device verifies the server’s certificate and marks it as trusted server after receiving the trust anchor. Here, if the system administrator opts not to validate the server, the server will instead pass on bootstrapping data in place of the “redirect-data”. In addition, the device will continue the bootstrapping process assuming the server is untrusted.

3. Artifact Validation

Artifact validation is important to ensure that the configuration files or software images used to provision network devices are authentic and have not been tampered with. Once the server validation is complete (or skipped), the bootstrap server will send the owner certificate, ownership voucher, and onboarding information to the device as bootstrapping data. Let’s discuss them closely to gain a better understanding.

Ownership Voucher (OV): The ownership voucher artifact validates the owner certificate to verify the identity of the device’s owner. The device manufacture signs the OV and provides it to the customer based on the request. To generate the OV, the customer must provide the pinned-domain-cert and serial number of the device to the Cisco MASA server.
Owner Certificate (OC): Owner Certificate is an X.509 certificate that binds an owner identity to a public key, which a device can use to validate signature over the conveyed information artifact. The owner certificate also holds all intermediate certificates that led to the “pinned-domain-cert” certificate specified in the ownership voucher, allowing the OV to validate the OC.
Conveyed Information/Onboarding Information: Onboarding information provides data necessary for a device to bootstrap itself and establish secure connections with other systems. Onboarding information specify details about the boot image a device must be running, an initial configuration the device must commit, and scripts that the device must successfully execute. The onboarding information must be signed by the device’s owner using OC.

Zero Trust is crucial when performing Day 0 provisioning

In addition to its many features, Secure ZTP goes beyond by offering audit trails and monitoring capabilities. This includes logging all provisioning events, configuration changes, and user actions. By monitoring ZTP activities, network administrators can quickly detect any suspicious activity and take appropriate action.

As we wrap up our discussion, it becomes clear that Zero Trust is also crucial when performing Day 0 provisioning, and Secure ZTP is the best way to ensure that zero trust principles are applied while performing Day 0 provisioning using a Do-It-Yourself (DIY) approach.

With the IOS-XE 17.11.1 release, users can now take advantage of the secure Zero Touch Provisioning (ZTP) capabilities with Catalyst 9000 series switches. This exciting feature aligns with the specifications outlined in RFC 8572, ensuring a secure and seamless provisioning experience. For more details about how to implement Secure ZTP, please refer the IOS-XE 17.11.1 Configuration Guide.

Keep Learning with these resources

Visit the Zero Touch Resource Hub
Learn more about Cisco IOS XE
Cisco IOS XE blogs
Zero Touch Provisioning blog

References

Cisco DNA Center: Early Results from Intent-based Networking
Security, Data Breaches Top Cause of Downtime in 2022
IBM – Cost of a Data Breach Report 2023

  Learn how Cisco Secure ZTP (Zero Touch Provisioning) helps you maintain security throughout the provisioning process with a proactive approach that employs robust authentication and encrypted communication channels to enhance the security posture of a network while Day 0 network automation is in place.  Read More Cisco Blogs 

As organizations build, modernize, and deploy business-critical applications including traditional and on-premise, hybrid applications, and modern, distributed (cloud native), they need more observability into their increasingly complex, dispersed workloads. Using multiple domain-specific monitoring, visibility, and observability tools can provide insights into individual domains, but it’s difficult to do at scale, and creates multiple sources of truth. How can organizations bring together enterprise workload details to the insights they need for better application experiences?

Breaking through layers of complexity with full stack observability

Cisco FSO Platform is an open and extensible, API-driven platform focused on OpenTelemetry and anchored on metrics, events, logs, and traces (MELT), providing AI/ML driven analytics. It brings data together from multiple domains including application, networking, infrastructure, security, cloud, sustainability, and business sources.

Cisco FSO platform also empowers a new observability ecosystem delivering relevant and impactful business insights through new use-cases and extensions. Cisc  flexible modules that help customers with specific business outcomes. One great example is the vSphere Observability and Data Modernization built by Cisco technology partner CloudFabrix.

CloudFabrix vSphere Observability and Data Modernization

Many organizations are utilizing the VMware vSphere enterprise workload platform for their business-critical applications. So CloudFabrix took the initiative to develop a module that helps them observe vSphere through Cisco FSO Platform for a more centralized, consolidated view.

“Most of our customers today are running hybrid environments, so we decided our first module should bring vSphere into Cisco FSO Platform, to make them “first class citizens” of FSO,” said Raja Penmetsa, CTO of CloudFabrix. “CloudFabrix collects the infastructure information underneath your application, translates into OTEL telemetry signals, and puts it into FSO.”

CloudFabrix enables organizations to acquire a vast amount of data and entity insights from across the technology stack make them easily accessible on Cisco FSO Platform, to support application or infrastructure modernization, performance impact analysis, and other use cases.

The module also enriches vSphere and VMware vRealize Operations data with their environment’s Kubernetes and infrastructure data.

CloudFabrix provides visibility and correlation into where services are running and how they are connected, including details about which Kubernetes service is running on a particular VMware host, on a specific VM.

Our technology partners like CloudFabrix say that Cisco FSO Platform has enabled them to hit the ground running as they commence work on expansion modules. Its rich modeling available right out of the box helps them build and bring new solutions to market fast, and rapidly fill use cases for specific domains and vertical use cases.

Seizing emerging market opportunities and use cases faster

“First and foremost, Cisco FSO Platform has done a really good job of enabling us to extend the platform,” said Penmetsa. “Based on our past experiences, something like this would have taken about six to 12 months to build. Working with the Cisco FSO Platform for the first time, it took us less than four weeks to build our vSphere Observability and Data Modernization module. And now that we’re familiar with the platform, we should be able to churn out future deliverables on our roadmap even faster.”

Watch the full interview between Raju Penmetsa, CTO, Cloudfabrix and Sunder Parameswaran, VP Product Management, Cisco FSO.

As Cisco continues to enhance the platform with additional tools and documentation for developers, CloudFabrix and other technology partners will gain access to even more robust support.

Together, Cisco FSO Platform and its partner modules deliver a comprehensive solution that scales with the pace of business, with easy extensibility everywhere, across the infrastructure and the application life cycle.

 

Learn more about  Cisco FSO Platform and the CloudFabrix Module

 

 

 

 

​

As organizations build, modernize, and deploy business-critical applications including traditional and on-premise, hybrid applications, and modern, distributed (cloud native), they need more observability into their increasingly complex, dispersed workloads. Using multiple domain-specific monitoring, visibility, and observability tools can provide insights into individual domains, but it’s difficult to do at scale, and creates multiple sources of truth. How can organizations bring together enterprise workload details to the insights they need for better application experiences?

Breaking through layers of complexity with full stack observability

Cisco FSO Platform is an open and extensible, API-driven platform focused on OpenTelemetry and anchored on metrics, events, logs, and traces (MELT), providing AI/ML driven analytics. It brings data together from multiple domains including application, networking, infrastructure, security, cloud, sustainability, and business sources.

Cisco FSO platform also empowers a new observability ecosystem delivering relevant and impactful business insights through new use-cases and extensions. Cisc  flexible modules that help customers with specific business outcomes. One great example is the vSphere Observability and Data Modernization built by Cisco technology partner CloudFabrix.

CloudFabrix vSphere Observability and Data Modernization

Many organizations are utilizing the VMware vSphere enterprise workload platform for their business-critical applications. So CloudFabrix took the initiative to develop a module that helps them observe vSphere through Cisco FSO Platform for a more centralized, consolidated view.

“Most of our customers today are running hybrid environments, so we decided our first module should bring vSphere into Cisco FSO Platform, to make them “first class citizens” of FSO,” said Raja Penmetsa, CTO of CloudFabrix. “CloudFabrix collects the infastructure information underneath your application, translates into OTEL telemetry signals, and puts it into FSO.”

CloudFabrix enables organizations to acquire a vast amount of data and entity insights from across the technology stack make them easily accessible on Cisco FSO Platform, to support application or infrastructure modernization, performance impact analysis, and other use cases.

The module also enriches vSphere and VMware vRealize Operations data with their environment’s Kubernetes and infrastructure data.

CloudFabrix provides visibility and correlation into where services are running and how they are connected, including details about which Kubernetes service is running on a particular VMware host, on a specific VM.

Our technology partners like CloudFabrix say that Cisco FSO Platform has enabled them to hit the ground running as they commence work on expansion modules. Its rich modeling available right out of the box helps them build and bring new solutions to market fast, and rapidly fill use cases for specific domains and vertical use cases.

Seizing emerging market opportunities and use cases faster

“First and foremost, Cisco FSO Platform has done a really good job of enabling us to extend the platform,” said Penmetsa. “Based on our past experiences, something like this would have taken about six to 12 months to build. Working with the Cisco FSO Platform for the first time, it took us less than four weeks to build our vSphere Observability and Data Modernization module. And now that we’re familiar with the platform, we should be able to churn out future deliverables on our roadmap even faster.”

Watch the full interview between Raju Penmetsa, CTO, Cloudfabrix and Sunder Parameswaran, VP Product Management, Cisco FSO.

As Cisco continues to enhance the platform with additional tools and documentation for developers, CloudFabrix and other technology partners will gain access to even more robust support.

Together, Cisco FSO Platform and its partner modules deliver a comprehensive solution that scales with the pace of business, with easy extensibility everywhere, across the infrastructure and the application life cycle.

Learn more about  Cisco FSO Platform and the CloudFabrix Module

  If you utilize VMware vSphere enterprise workload platform for your business-critical applications, take a look at the CloudFabrix module that helps you observe vSphere through Cisco FSO Platform for a more centralized, consolidated view.  Read More Cisco Blogs 

A deployment guide for wireless ThousandEyes agents deployed to monitor the Black Hat 2023 conference by Adam Kilgore & Ryan MacLennan

ThousandEyes (TE) Black Hat 2023 Deployment Guide

This guide documents the setup and installation procedures used to deploy ThousandEyes at Black Hat 2023. This document covers configuration of the TE agent on Raspberry Pi 4, including both wired and wireless configurations. Needed hardware for the various setup functions are also listed.

Limitations of Deploying ThousandEyes Enterprise Agents on Raspberry Pi

Please note that TE Enterprise Agents will only show data for layer 3 hops, and will not show wireless access point data. This applies to deploying the TE Enterprise Agent on the Raspberry Pi and configuring it for wireless. The only TE agent that can provide wireless AP data is the TE Endpoint Agent, which is intended to be installed on laptops and personal devices.

Hardware Checklist for the Conference

There are several different hardware contingencies that must be accounted for before conference setup can take place.

A keyboard that can connect to the USB port on the Raspberry Pi
Laptops that can connect to the Pi Micro SD for TE agent provisioning. We used several different solutions for this:

An SD card to Micro SD adapter, which connected to the SD reader on a Mac laptop (the SD reader is not available on all models)
A Vivitar Micro SD to USB-C adapter, which connected to the USB-C port on a Mac laptop
A Micro SD to USB adapter that came with the Raspberry Pi 4 Cana Kit. This was connected to a USB to USB-C Mac adapter

Screens that can connect to the Raspberry Pi Micro HDMI port. This can be as simple as a normal HDMI monitor that connects to the Pi via HDMI to Micro HDMI, but these were not available during the setup days of the conference. We used two solutions:

A portable monitor that connected to the Pi via an HDMI to Micro HDMI cable
An adapter that connected to the Pi via Micro HDMI, connected to a Mac HDMI adapter, with the Mac HDMI adapter connected to a USB-C port on the Mac laptop

Pi Hardware Assembly

Black Hat 2023 Pi 4 Kits

10x Raspberry Pi 4 Cana Kits, each with the following:

32GB Micro SD card
Micro SD to USB adapter
Case
Heatsink
Fan

Example of built Pi with heatsinks and fan attached:

Note that we had overheating issues with the fan set to pull air out of the case. Orienting the fan to blow into the Pi case was more effective for our deployment.

Provision the ThousandEyes Image onto the Pi Micro SD

The Raspberry Pi can read Micro SD cards. The installation procedure is to download the TE Pi installer from the TE GUI, connect the Micro SDs to the provisioning laptops, and then use balenaEtcher to install the TE image to the Micro SD.

Retrieving the TE Agent Image & Documentation

From the ThousandEyes GUI, navigate to Cloud & Enterprise Agents > Agent Settings.

Click on Add New Enterprise Agent.

Locate the entry for Raspberry Pi 4 and click the Download – IMG button. The Installation Guide for the Pi install is also available from this page.

Connect the Micro SD(s) to Provisioning Laptops

If the provisioning laptops do not have a Micro SD card reader then adapter solutions will be needed to install the TE image to the Micro SD. For Black Hat 2023, we used a combination of the following hardware with Mac laptops.

2x Vivitar Mac USB-C to Micro SD adapter

1x Mac USB-C to USB adapter + 1x USB Micro SD Card Reader (included with some Pi bundles—the USB has an additional slot for the Micro SD card, which then plugs into a USB port—in this case, the Mac adapter shown).

1x Micro SD to SD adapter

Install the TE Image to Micro SDs

We used balenaEtcher to write the TE images to Micro SD, as covered in the TE Pi deployment guide.

Note that Etcher can write multiple images to multiple Micro SDs at once, your only limit is the number of Micro SD cards that can be connected to the provisioning laptops (USB-C, Micro SD card reader, USB port, etc.) at once.

Optional: Use Scripts to Provision Wireless that will Persist Across Reboots

Creating the wireless configuration for the Pi can occur either before or after booting the Pi for the first time with the Micro SD image that was configured in the prior section. If you would like to create the wireless configuration before booting the Pi, please follow the steps in this section. Alternatively, if wireless is not being used or wireless configuration can occur after booting the Pi, this section can safely be skipped. Please note that configuring wireless after booting the Pi will require enabling SSH on the TE agent, along with any requisite firewall rules to reach the Pi over port 22.

Also note that this section requires specific hardware and software, as covered in the next paragraphs. If the needed hardware and software is not available, the wireless configuration can still be accomplished using manual procedures given in later sections.

Before beginning the configuration, note that the SSID and SSID password must be hard coded. If the TE agent cannot connect to the hardcoded SSID after reboot, the wireless connection will fail. If the hardcoded SSID and password need to be changed after the initial setup, it will be necessary to (re)enable SSH on the TE agent.

Also note that the TE agent file system must be mounted onto a laptop to perform these steps. At Black Hat 2023, we were only able to accomplish this step using a Linux OS. Attempts to mount the Micro SD on Mac did not correctly mount the TE agent partitions.

Log in to a Linux system and connect a Micro SD with the TE agent installed (this procedure was covered in the prior section). Use the Micro SD connection methodology provided at the start of the guide.
Run the df command to verify the mounted partitions. The below screenshot shows the df output before and after connecting the Micro SD card, with the two mounted partitions visible after running the 2nd df command. Note the partition paths highlighted in red, as they will be needed to create the doit.sh script in Step 5.

Create a file named rc.local in a chosen directory. The specific name ‘rc.local’ is necessary for the script to run during the TE agent bootup process.
Create the script file:

# vim rc.local

Type ‘i’ to edit, and type or paste the following:

#!/bin/bash
/configure_te_pi.sh
exit 0

Type ‘:’ then ‘wq!’ to save the changes and exit the file.
Create the configure_te_pi.sh script in the same directory as the rc.local file. Replace the <SSID> and <SSID password> fields in the below script with the SSID and PSK that the Pi will connect to. The SSID and PSK can also be changed later, but this will require enabling SSH on the TE agent and then connecting to it.
Create the script file:

# vim configure_te_pi.sh

Type ‘i’ to edit, and type or paste the following:

#!/bin/bash
apt-get update -y
apt-get install te-va-unlock -y
apt-get install net-tools ifmetric wireless-tools -y
ifconfig wlan0 up
wpa_passphrase <SSID> <SSID password> > /etc/wpa_supplicant.conf
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf -D wext
ifmetric eth0 200
sed -i ‘1iauto wlan0niface wlan0 inet dhcp’ /etc/network/interfaces
dhclient wlan0
systemctl mask apt-news.service
systemctl mask esm-cache.service

Type ‘:’ then ‘wq!’ to save the changes and exit the file.
Create the doit.sh script in the same directory as the prior two scripts. Note that the paths to the /writeable and /system-boot mounts will need to be filled in with the paths collected in Step 1.
Create the script file:

# vim doit.sh

Type ‘i’ to edit, and type or paste the following, modifying the paths to match the output in Step 1:

#!/bin/bash
cp rc.local /media/<username>/writeable/etc
cp configure_te_pi.sh /media/<username>/writeable/etc
chmod +x /media/<username>/writeable/configure_te_pi.sh
chmod +x / media/<username>/writeable/etc/rc.local
umount /media/<username>/system-boot
umount /media/<username>/writable

Type ‘:’ then ‘wq!’ to save the changes and exit the file.

An example of the script created for the output in Step 1 is provided below.

#!/bin/bash
cp ~iredden/rc.local /media/iredden/writable/etc
cp ~iredden/configure_te_pi.sh /media/iredden/writable
chmod +x /media/iredden/writable/configure_te_pi.sh
chmod +x /media/iredden/writable/etc/rc.local
umount /media/iredden/system-boot
umount /media/iredden/writable

Run the doit.sh script. The doit.sh script will perform the configuration and then unmount the partitions, as shown below.

With the above steps complete, the Micro SD card can be removed from the laptop and connected to the Raspberry Pi. The Pi will execute the scripts during bootup and connect to wireless, if the SSID configured is reachable and the PSK is correct. Please see the next section for setup instructions and other considerations.

Also note that a wired connection to the Pi is still recommended for initial setup. Although the Pi TE agent can successfully pull a wireless IP address during initial bootup, the TE agent will not display a wireless IP address on a connected monitor, only a wired one.

Initial Setup of TE Agent

The Provision the ThousandEyes (TE) Image onto the Pi Micro SD section of this guide covered how to provision TE agents onto Micro SD cards. An optional section covering how to script wireless capability onto the TE agent image was also provided in the Use Scripts to Configure Wireless that will Persist Across Reboots section. This section will cover how to use the TE agent on the Micro SD card to perform initial setup.

Connect Micro SD to Raspberry Pi

Once the TE image has been installed on the Micro SD, connect the Micro SD to the Raspberry Pi.

Connect an Ethernet Cable to the Pi

It is necessary to know the IP address assigned to a TE agent after initial boot in order to complete initial registration. While a Pi with a wireless configured TE agent can pull a wireless IP during its initial bootup, it will not display the wireless IP on a connected monitor, only a wired IP address. For this reason, it is still recommended to connect an ethernet cable to a wireless Pi before initial power on and use the assigned eth0 IP to perform initial setup. The only alternative to a wired connection on initial boot is to identify the wireless IP address of the TE agent by means other than a connected monitor, such as via direct access to the wireless AP that the TE agent connects to.

Connect Keyboard and Monitor to Raspberry Pi

Two pieces of hardware are used to perform the initial setup of the Raspberry Pi TE agent: (1) a keyboard that can connect to the Pi via USB, and (2) a screen that can connect to the Pi via Micro HDMI. Initial setup can be performed without the keyboard if (a) an IP address is assigned to the TE agent via DHCP, and (b) the DHCP address can be identified via the connected screen or by other means. Similarly, initial setup can be completed without the screen if there is another way to identify the DHCP IP address assigned to the TE agent, such as admin access to the switch connected to the TE agent.

Despite the above alternatives, a connected keyboard and monitor are recommended for ease of deployment and any needed troubleshooting.

For the keyboard, we used a Bluetooth keyboard with a detached USB fob, but any keyboard that can connect to the Pi via USB will work.

For the screen, we used two separate hardware solutions.

The first solution is a Kenowa portable monitor with an HDMI to Micro HDMI cable. The HDMI side of the cable connects to the Kenowa, and the Micro HDMI side of the cable connects to the Raspberry Pi.

The second solution we used is an Elgato adapter that sits between the Raspberry Pi and a Mac laptop. The Pi connects to the Elgato via a micro HDMI to HDMI cable. The Elgato uses a regular HDMI cable to connect to the HDMI side of a Mac HDMI to USB-C adapter, with the USB-C side of the Mac adapter connecting directly to a Mac laptop.

Confirm Pi Hardware Connections

With the hardware configurations in the prior sections complete (i.e. the Pi is connected to an ethernet port, monitor, and keyboard), connect the Pi power adapter to begin initial boot.

Connect to the TE Agent GUI

Once the Pi is powered on, the TE agent install will take a few minutes to run (more if internet access is not available). Once the bootup is complete, the Pi will display a message if an IP address was assigned to the ethernet port via DHCP.

If no IP address was assigned to the ethernet port via DHCP, the TE agent will display a different message stating no IP address is configured. If this is the case, it will be necessary to either manually assign an IP address to the ethernet port using a keyboard, or identify the IP address of the wireless interface using admin access to the connected wireless AP.

Once the IP address of the TE agent is known, connect to the IP via HTTPS and sign in with username admin and password welcome.

Change the default password to something more secure and click Change Password.

The GUI should automatically take you to Agent in the left menu.

Keeping the ThousandEyes Physical Appliance GUI open, create a new tab in your browser and connect to the ThousandEyes web GUI, then click on Cloud Enterprise Agents > Agent Settings.

Click on the Add New Enterprise Agent

Copy the Access Group Token. Note: the Copy button may not work, in which case click the eye icon to show the token and copy it manually.

Return to the TE agent GUI and paste the Account Group Token into the Agent Click Continue.

The GUI will automatically load the Review Appliance Status and Diagnostics fields should change to green as the registration completes. Click the Complete button.

Click on the Network tab.

Configure the hostname for the agent.

Scroll down to the DNS configuration and enter the internal and External DNS servers. By Default, the agent will use the 208.67.222.222 and 208.67.222.220 external Umbrella DNS servers. These can be moved to DNS server 3 and 4 if internal DNS servers are preferred.

Scroll down and click on the Save Changes Note that the Applying network settings pop-up may get stuck. Reload the page to clear it.
Connect to the ThousandEyes web GUI and confirm that the agent shows as active under Enterprise Agents. For additional guidance, please see the section Validate Wireless Interface in the ThousandEyes Web GUI.

Manual Wireless Config Track

The Raspberry Pi TE agent can only connect via a wired connection using the default installation. If wireless is needed for the deployment but the necessary hardware/software was not available for the Optional: Use Scripts to Configure Wireless that will Persist Across Reboots section, or if the TE agent has already been deployed, the series of manual configuration steps starting in this section can be used to configure persistent wireless on a Pi TE agent.

Manual Wireless Config Track: Connect to the TE Agent via SSH

Documentation for connecting to a Pi TE agent via SSH can be found in this guide.

Manual Wireless Config Track: Jailbreak the TE Agent (to allow Wireless config)

Jailbreaking the TE agent is necessary for escalating to root and performing the commands necessary to build the scripts for the wireless configuration.

We should update the Ubuntu OS before running the jailbreak command. Connect to the TE agent via SSH, then run the following two commands to first update the OS and then jailbreak the install:

$ sudo apt-get update
$ sudo apt-get install te-va-unlock

With the TE agent successfully jailbroken, we can now escalate to root and perform the rest of the configuration.

Manual Wireless Config Track: Configure Wireless that will Persist Across Reboots using only the Pi

Before beginning this section, note that the SSID and SSID password must be hard coded. If the TE agent cannot connect to the hardcoded SSID after reboot, the wireless connection will fail. If the hardcoded SSID and password need to be changed after the initial setup, it will be necessary to (re)enable SSH on the TE agent and connect to it over SSH.

Escalate to root and create a file named rc.local in the /etc directory. The name is necessary for the script to run during the TE agent bootup process.

Create the script file:

$ sudo su
# cd /etc
# vim rc.local

Type ‘i’ to edit, and type or paste the following:

#!/bin/bash
/configure_te_pi.sh
exit 0

Type ‘:’ then ‘wq!’ to save the changes and exit the file. 
Run the following to set the script permissions:

# chmod +x /etc/rc.local

Create the configure_te_pi.sh script in the top level / directory. Replace the <SSID> and <SSID password> fields in the below script with the correct values. The SSID and password can also be changed later, but this will require a later SSH connection to the agent.
Create the script file:

# cd /
# vim configure_te_pi.sh

Type ‘i’ to edit, and type or paste the following, replacing the <SSID> and <SSID password> fields with the correct values:

#!/bin/bash
apt-get update -y
apt-get install te-va-unlock -y
apt-get install net-tools ifmetric wireless-tools -y
ifconfig wlan0 up
wpa_passphrase <SSID> <SSID password> > /etc/wpa_supplicant.conf
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf -D wext
ifmetric eth0 200
sed -i ‘1iauto wlan0niface wlan0 inet dhcp’ /etc/network/interfaces
dhclient wlan0
systemctl mask apt-news.service
systemctl mask esm-cache.service

Type ‘:’ then ‘wq!’ to save the changes and exit the file.
Run the following to set the script permissions:

# chmod +x /configure_te_pi.sh
Reboot the Pi and test to confirm that the wireless NIC can connect to the ThousandEyes web account.

Validate Wireless Interface in the ThousandEyes Web GUI

Confirm Wired and Wireless IPs

From the ThousandEyes web GUI, navigate to Cloud & Enterprise Agents > Agent Settings.

Identify a registered TE agent with a wireless interface, then click on it.

Locate the General Information The private IP address of the TE agent will be displayed, which is the IP address that the TE agent used to connect to the ThousandEyes web manager. If the TE agent has both an ethernet and wireless port, you can click the System Information link to confirm which IP address is assigned to each interface.

Confirm Wired and Wireless Interfaces for Tests

Navigate to Cloud & Enterprise Agents > Test Settings.

Expand one of the tests and then expand the Agents

Click on the Enterprise button and expand the dropdown next to an agent with wireless enabled. If the agent has both a wired and a wireless interface enabled, they will both be displayed as in the screenshot below.

About Black Hat

For 26 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: BlackHat.com. Black Hat is brought to you by Informa Tech.

Appendix

GitHub with scripts

ThousandEyes Wireless Setup GitHub

ThousandEyes Ports and Protocols

For TE agents to operate correctly, they will need to both connect to ThousandEyes over the internet and have firewall ACL access to perform any configured tests. Refer to the following document for lists of ports and protocols per region.

https://docs.thousandeyes.com/product-documentation/global-vantage-points/enterprise-agents/configuring/firewall-configuration-for-enterprise-agents

Script Functions Explained

Scripts to enable wireless functionality of the Raspberry Pi TE agent on startup are included in the Optional: Use Scripts to Provision Wireless that will Persist Across Reboots and Manual Wireless Config Track sections of this guide. This appendix section explains the script functionality line by line.

rc.local – this script will run as part of the TE agent startup process.

#!/bin/bash – initialize the script to run using bash
/configure_te_pi.sh – run the configure_te_pi.sh script
exit 0 – exit

configure_te_pi.sh – this script will configure the TE agent for wireless functionality

#!/bin/bash – initialize the script
apt-get update -y – update the underlying Ubuntu OS
apt-get install te-va-unlock -y – jailbreak the TE agent to allow additional sudo access
apt-get install net-tools ifmetric wireless-tools -y – install tools to allow wireless functionality
ifconfig wlan0 up – bring up a wireless LAN interface
wpa_passphrase <SSID> <SSID password> > /etc/wpa_supplicant.conf – configure SSID and PSK values for the wireless connection and write them to the wpa_supplicant.conf file
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf -D wext – imports the wpa_supplicant file and configures the wireless LAN interface to use the wext driver
ifmetric eth0 200 – set the wired interface to have a lower priority than the wireless interface
sed -i ‘1iauto wlan0niface wlan0 inet dhcp’ /etc/network/interfaces – configure DHCP for the wireless interface
dhclient wlan0 – enable DHCP for the wireless interface
systemctl mask apt-news.service – prevent apt-news.service from running. See the Wireless TE Agent Update Failure section for more information
systemctl mask esm-cache.service – prevent the esm-cache.service from running. See the Wireless TE Agent Update Failure section for more information

Wireless TE Agent Update Failure

During testing of the wireless TE agent for Raspberry Pi, it was noticed that the agents lost connectivity with the ThousandEyes web manager after a variable amount of time. Troubleshooting multiple failed agents identified logs associated with updates to the apt-news.service and esm-cache.service processes immediately before failure of the wlan:

syslog

Aug  6 02:17:04 registration-wireless systemd[1]: Started PackageKit Daemon.
Aug  6 02:17:20 registration-wireless systemd[1]: Starting Update APT News…
Aug  6 02:17:20 registration-wireless systemd[1]: Starting Update the local ESM caches…
Aug  6 02:17:21 registration-wireless systemd[1]: apt-news.service: Succeeded.
Aug  6 02:17:21 registration-wireless systemd[1]: Finished Update APT News.
Aug  6 02:17:22 registration-wireless systemd[1]: esm-cache.service: Succeeded.
Aug  6 02:17:22 registration-wireless systemd[1]: Finished Update the local ESM caches.
Aug  6 02:18:38 registration-wireless systemd-networkd[667]: wlan0: Lost carrier

More information on the two services can be found here:

https://askubuntu.com/questions/1452519/what-are-the-services-apt-news-and-esm-cache-and-how-do-i-disable-them

Masking the services so they are not run by systemd resolved the failure of wlan0:

$ sudo systemctl mask apt-news.service
$ sudo systemctl mask esm-cache.service

The above commands were then added to the startup scripts.

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

Instagram
Facebook
Twitter
LinkedIn

​

A deployment guide for wireless ThousandEyes agents deployed to monitor the Black Hat 2023 conference by Adam Kilgore & Ryan MacLennan

ThousandEyes (TE) Black Hat 2023 Deployment Guide

This guide documents the setup and installation procedures used to deploy ThousandEyes at Black Hat 2023. This document covers configuration of the TE agent on Raspberry Pi 4, including both wired and wireless configurations. Needed hardware for the various setup functions are also listed.

Limitations of Deploying ThousandEyes Enterprise Agents on Raspberry Pi

Please note that TE Enterprise Agents will only show data for layer 3 hops, and will not show wireless access point data. This applies to deploying the TE Enterprise Agent on the Raspberry Pi and configuring it for wireless. The only TE agent that can provide wireless AP data is the TE Endpoint Agent, which is intended to be installed on laptops and personal devices.

Hardware Checklist for the Conference

There are several different hardware contingencies that must be accounted for before conference setup can take place.

A keyboard that can connect to the USB port on the Raspberry Pi
Laptops that can connect to the Pi Micro SD for TE agent provisioning. We used several different solutions for this:
An SD card to Micro SD adapter, which connected to the SD reader on a Mac laptop (the SD reader is not available on all models)
A Vivitar Micro SD to USB-C adapter, which connected to the USB-C port on a Mac laptop
A Micro SD to USB adapter that came with the Raspberry Pi 4 Cana Kit. This was connected to a USB to USB-C Mac adapter

Screens that can connect to the Raspberry Pi Micro HDMI port. This can be as simple as a normal HDMI monitor that connects to the Pi via HDMI to Micro HDMI, but these were not available during the setup days of the conference. We used two solutions:
A portable monitor that connected to the Pi via an HDMI to Micro HDMI cable
An adapter that connected to the Pi via Micro HDMI, connected to a Mac HDMI adapter, with the Mac HDMI adapter connected to a USB-C port on the Mac laptop

Pi Hardware Assembly

Black Hat 2023 Pi 4 Kits

10x Raspberry Pi 4 Cana Kits, each with the following:

32GB Micro SD card
Micro SD to USB adapter
Case
Heatsink
Fan

Example of built Pi with heatsinks and fan attached:

Note that we had overheating issues with the fan set to pull air out of the case. Orienting the fan to blow into the Pi case was more effective for our deployment.

Provision the ThousandEyes Image onto the Pi Micro SD

The Raspberry Pi can read Micro SD cards. The installation procedure is to download the TE Pi installer from the TE GUI, connect the Micro SDs to the provisioning laptops, and then use balenaEtcher to install the TE image to the Micro SD.

Retrieving the TE Agent Image & Documentation

From the ThousandEyes GUI, navigate to Cloud & Enterprise Agents > Agent Settings.

Click on Add New Enterprise Agent.

Locate the entry for Raspberry Pi 4 and click the Download – IMG button. The Installation Guide for the Pi install is also available from this page.

Connect the Micro SD(s) to Provisioning Laptops

If the provisioning laptops do not have a Micro SD card reader then adapter solutions will be needed to install the TE image to the Micro SD. For Black Hat 2023, we used a combination of the following hardware with Mac laptops.

2x Vivitar Mac USB-C to Micro SD adapter

1x Mac USB-C to USB adapter + 1x USB Micro SD Card Reader (included with some Pi bundles—the USB has an additional slot for the Micro SD card, which then plugs into a USB port—in this case, the Mac adapter shown).

1x Micro SD to SD adapter

Install the TE Image to Micro SDs

We used balenaEtcher to write the TE images to Micro SD, as covered in the TE Pi deployment guide.

Note that Etcher can write multiple images to multiple Micro SDs at once, your only limit is the number of Micro SD cards that can be connected to the provisioning laptops (USB-C, Micro SD card reader, USB port, etc.) at once.

Optional: Use Scripts to Provision Wireless that will Persist Across Reboots

Creating the wireless configuration for the Pi can occur either before or after booting the Pi for the first time with the Micro SD image that was configured in the prior section. If you would like to create the wireless configuration before booting the Pi, please follow the steps in this section. Alternatively, if wireless is not being used or wireless configuration can occur after booting the Pi, this section can safely be skipped. Please note that configuring wireless after booting the Pi will require enabling SSH on the TE agent, along with any requisite firewall rules to reach the Pi over port 22.

Also note that this section requires specific hardware and software, as covered in the next paragraphs. If the needed hardware and software is not available, the wireless configuration can still be accomplished using manual procedures given in later sections.

Before beginning the configuration, note that the SSID and SSID password must be hard coded. If the TE agent cannot connect to the hardcoded SSID after reboot, the wireless connection will fail. If the hardcoded SSID and password need to be changed after the initial setup, it will be necessary to (re)enable SSH on the TE agent.

Also note that the TE agent file system must be mounted onto a laptop to perform these steps. At Black Hat 2023, we were only able to accomplish this step using a Linux OS. Attempts to mount the Micro SD on Mac did not correctly mount the TE agent partitions.

Log in to a Linux system and connect a Micro SD with the TE agent installed (this procedure was covered in the prior section). Use the Micro SD connection methodology provided at the start of the guide.
Run the df command to verify the mounted partitions. The below screenshot shows the df output before and after connecting the Micro SD card, with the two mounted partitions visible after running the 2nd df command. Note the partition paths highlighted in red, as they will be needed to create the doit.sh script in Step 5.

Create a file named rc.local in a chosen directory. The specific name ‘rc.local’ is necessary for the script to run during the TE agent bootup process.
Create the script file:

# vim rc.local

Type ‘i’ to edit, and type or paste the following:

#!/bin/bash
/configure_te_pi.sh
exit 0

Type ‘:’ then ‘wq!’ to save the changes and exit the file.
Create the configure_te_pi.sh script in the same directory as the rc.local file. Replace the <SSID> and <SSID password> fields in the below script with the SSID and PSK that the Pi will connect to. The SSID and PSK can also be changed later, but this will require enabling SSH on the TE agent and then connecting to it.
Create the script file:

# vim configure_te_pi.sh

Type ‘i’ to edit, and type or paste the following:

#!/bin/bash
apt-get update -y
apt-get install te-va-unlock -y
apt-get install net-tools ifmetric wireless-tools -y
ifconfig wlan0 up
wpa_passphrase <SSID> <SSID password> > /etc/wpa_supplicant.conf
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf -D wext
ifmetric eth0 200
sed -i ‘1iauto wlan0niface wlan0 inet dhcp’ /etc/network/interfaces
dhclient wlan0
systemctl mask apt-news.service
systemctl mask esm-cache.service

Type ‘:’ then ‘wq!’ to save the changes and exit the file.
Create the doit.sh script in the same directory as the prior two scripts. Note that the paths to the /writeable and /system-boot mounts will need to be filled in with the paths collected in Step 1.
Create the script file:

# vim doit.sh

Type ‘i’ to edit, and type or paste the following, modifying the paths to match the output in Step 1:

#!/bin/bash
cp rc.local /media/<username>/writeable/etc
cp configure_te_pi.sh /media/<username>/writeable/etc
chmod +x /media/<username>/writeable/configure_te_pi.sh
chmod +x / media/<username>/writeable/etc/rc.local
umount /media/<username>/system-boot
umount /media/<username>/writable

Type ‘:’ then ‘wq!’ to save the changes and exit the file.

An example of the script created for the output in Step 1 is provided below.

#!/bin/bash
cp ~iredden/rc.local /media/iredden/writable/etc
cp ~iredden/configure_te_pi.sh /media/iredden/writable
chmod +x /media/iredden/writable/configure_te_pi.sh
chmod +x /media/iredden/writable/etc/rc.local
umount /media/iredden/system-boot
umount /media/iredden/writable

Run the doit.sh script. The doit.sh script will perform the configuration and then unmount the partitions, as shown below.

With the above steps complete, the Micro SD card can be removed from the laptop and connected to the Raspberry Pi. The Pi will execute the scripts during bootup and connect to wireless, if the SSID configured is reachable and the PSK is correct. Please see the next section for setup instructions and other considerations.

Also note that a wired connection to the Pi is still recommended for initial setup. Although the Pi TE agent can successfully pull a wireless IP address during initial bootup, the TE agent will not display a wireless IP address on a connected monitor, only a wired one.

Initial Setup of TE Agent

The Provision the ThousandEyes (TE) Image onto the Pi Micro SD section of this guide covered how to provision TE agents onto Micro SD cards. An optional section covering how to script wireless capability onto the TE agent image was also provided in the Use Scripts to Configure Wireless that will Persist Across Reboots section. This section will cover how to use the TE agent on the Micro SD card to perform initial setup.

Connect Micro SD to Raspberry Pi

Once the TE image has been installed on the Micro SD, connect the Micro SD to the Raspberry Pi.

Connect an Ethernet Cable to the Pi

It is necessary to know the IP address assigned to a TE agent after initial boot in order to complete initial registration. While a Pi with a wireless configured TE agent can pull a wireless IP during its initial bootup, it will not display the wireless IP on a connected monitor, only a wired IP address. For this reason, it is still recommended to connect an ethernet cable to a wireless Pi before initial power on and use the assigned eth0 IP to perform initial setup. The only alternative to a wired connection on initial boot is to identify the wireless IP address of the TE agent by means other than a connected monitor, such as via direct access to the wireless AP that the TE agent connects to.

Connect Keyboard and Monitor to Raspberry Pi

Two pieces of hardware are used to perform the initial setup of the Raspberry Pi TE agent: (1) a keyboard that can connect to the Pi via USB, and (2) a screen that can connect to the Pi via Micro HDMI. Initial setup can be performed without the keyboard if (a) an IP address is assigned to the TE agent via DHCP, and (b) the DHCP address can be identified via the connected screen or by other means. Similarly, initial setup can be completed without the screen if there is another way to identify the DHCP IP address assigned to the TE agent, such as admin access to the switch connected to the TE agent.

Despite the above alternatives, a connected keyboard and monitor are recommended for ease of deployment and any needed troubleshooting.

For the keyboard, we used a Bluetooth keyboard with a detached USB fob, but any keyboard that can connect to the Pi via USB will work.

For the screen, we used two separate hardware solutions.

The first solution is a Kenowa portable monitor with an HDMI to Micro HDMI cable. The HDMI side of the cable connects to the Kenowa, and the Micro HDMI side of the cable connects to the Raspberry Pi.

The second solution we used is an Elgato adapter that sits between the Raspberry Pi and a Mac laptop. The Pi connects to the Elgato via a micro HDMI to HDMI cable. The Elgato uses a regular HDMI cable to connect to the HDMI side of a Mac HDMI to USB-C adapter, with the USB-C side of the Mac adapter connecting directly to a Mac laptop.

Confirm Pi Hardware Connections

With the hardware configurations in the prior sections complete (i.e. the Pi is connected to an ethernet port, monitor, and keyboard), connect the Pi power adapter to begin initial boot.

Connect to the TE Agent GUI

Once the Pi is powered on, the TE agent install will take a few minutes to run (more if internet access is not available). Once the bootup is complete, the Pi will display a message if an IP address was assigned to the ethernet port via DHCP.

If no IP address was assigned to the ethernet port via DHCP, the TE agent will display a different message stating no IP address is configured. If this is the case, it will be necessary to either manually assign an IP address to the ethernet port using a keyboard, or identify the IP address of the wireless interface using admin access to the connected wireless AP.

Once the IP address of the TE agent is known, connect to the IP via HTTPS and sign in with username admin and password welcome.

Change the default password to something more secure and click Change Password.

The GUI should automatically take you to Agent in the left menu.

Keeping the ThousandEyes Physical Appliance GUI open, create a new tab in your browser and connect to the ThousandEyes web GUI, then click on Cloud Enterprise Agents > Agent Settings.

Click on the Add New Enterprise Agent

Copy the Access Group Token. Note: the Copy button may not work, in which case click the eye icon to show the token and copy it manually.

Return to the TE agent GUI and paste the Account Group Token into the Agent Click Continue.

The GUI will automatically load the Review Appliance Status and Diagnostics fields should change to green as the registration completes. Click the Complete button.

Click on the Network tab.

Configure the hostname for the agent.

Scroll down to the DNS configuration and enter the internal and External DNS servers. By Default, the agent will use the 208.67.222.222 and 208.67.222.220 external Umbrella DNS servers. These can be moved to DNS server 3 and 4 if internal DNS servers are preferred.

Scroll down and click on the Save Changes Note that the Applying network settings pop-up may get stuck. Reload the page to clear it.
Connect to the ThousandEyes web GUI and confirm that the agent shows as active under Enterprise Agents. For additional guidance, please see the section Validate Wireless Interface in the ThousandEyes Web GUI.

Manual Wireless Config Track

The Raspberry Pi TE agent can only connect via a wired connection using the default installation. If wireless is needed for the deployment but the necessary hardware/software was not available for the Optional: Use Scripts to Configure Wireless that will Persist Across Reboots section, or if the TE agent has already been deployed, the series of manual configuration steps starting in this section can be used to configure persistent wireless on a Pi TE agent.

Manual Wireless Config Track: Connect to the TE Agent via SSH

Documentation for connecting to a Pi TE agent via SSH can be found in this guide.

Manual Wireless Config Track: Jailbreak the TE Agent (to allow Wireless config)

Jailbreaking the TE agent is necessary for escalating to root and performing the commands necessary to build the scripts for the wireless configuration.

We should update the Ubuntu OS before running the jailbreak command. Connect to the TE agent via SSH, then run the following two commands to first update the OS and then jailbreak the install:

$ sudo apt-get update
$ sudo apt-get install te-va-unlock

With the TE agent successfully jailbroken, we can now escalate to root and perform the rest of the configuration.

Manual Wireless Config Track: Configure Wireless that will Persist Across Reboots using only the Pi

Before beginning this section, note that the SSID and SSID password must be hard coded. If the TE agent cannot connect to the hardcoded SSID after reboot, the wireless connection will fail. If the hardcoded SSID and password need to be changed after the initial setup, it will be necessary to (re)enable SSH on the TE agent and connect to it over SSH.

Escalate to root and create a file named rc.local in the /etc directory. The name is necessary for the script to run during the TE agent bootup process.

Create the script file:

$ sudo su
# cd /etc
# vim rc.local

Type ‘i’ to edit, and type or paste the following:

#!/bin/bash
/configure_te_pi.sh
exit 0

Type ‘:’ then ‘wq!’ to save the changes and exit the file. 
Run the following to set the script permissions:

# chmod +x /etc/rc.local

Create the configure_te_pi.sh script in the top level / directory. Replace the <SSID> and <SSID password> fields in the below script with the correct values. The SSID and password can also be changed later, but this will require a later SSH connection to the agent.
Create the script file:

# cd /
# vim configure_te_pi.sh

Type ‘i’ to edit, and type or paste the following, replacing the <SSID> and <SSID password> fields with the correct values:

#!/bin/bash
apt-get update -y
apt-get install te-va-unlock -y
apt-get install net-tools ifmetric wireless-tools -y
ifconfig wlan0 up
wpa_passphrase <SSID> <SSID password> > /etc/wpa_supplicant.conf
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf -D wext
ifmetric eth0 200
sed -i ‘1iauto wlan0niface wlan0 inet dhcp’ /etc/network/interfaces
dhclient wlan0
systemctl mask apt-news.service
systemctl mask esm-cache.service

Type ‘:’ then ‘wq!’ to save the changes and exit the file.
Run the following to set the script permissions:

# chmod +x /configure_te_pi.sh
Reboot the Pi and test to confirm that the wireless NIC can connect to the ThousandEyes web account.

Validate Wireless Interface in the ThousandEyes Web GUI

Confirm Wired and Wireless IPs

From the ThousandEyes web GUI, navigate to Cloud & Enterprise Agents > Agent Settings.

Identify a registered TE agent with a wireless interface, then click on it.

Locate the General Information The private IP address of the TE agent will be displayed, which is the IP address that the TE agent used to connect to the ThousandEyes web manager. If the TE agent has both an ethernet and wireless port, you can click the System Information link to confirm which IP address is assigned to each interface.

Confirm Wired and Wireless Interfaces for Tests

Navigate to Cloud & Enterprise Agents > Test Settings.

Expand one of the tests and then expand the Agents

Click on the Enterprise button and expand the dropdown next to an agent with wireless enabled. If the agent has both a wired and a wireless interface enabled, they will both be displayed as in the screenshot below.

About Black Hat

For 26 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: BlackHat.com. Black Hat is brought to you by Informa Tech.

Appendix

GitHub with scripts

ThousandEyes Wireless Setup GitHub

ThousandEyes Ports and Protocols

For TE agents to operate correctly, they will need to both connect to ThousandEyes over the internet and have firewall ACL access to perform any configured tests. Refer to the following document for lists of ports and protocols per region.

https://docs.thousandeyes.com/product-documentation/global-vantage-points/enterprise-agents/configuring/firewall-configuration-for-enterprise-agents

Script Functions Explained

Scripts to enable wireless functionality of the Raspberry Pi TE agent on startup are included in the Optional: Use Scripts to Provision Wireless that will Persist Across Reboots and Manual Wireless Config Track sections of this guide. This appendix section explains the script functionality line by line.

rc.local – this script will run as part of the TE agent startup process.

#!/bin/bash – initialize the script to run using bash
/configure_te_pi.sh – run the configure_te_pi.sh script
exit 0 – exit

configure_te_pi.sh – this script will configure the TE agent for wireless functionality

#!/bin/bash – initialize the script
apt-get update -y – update the underlying Ubuntu OS
apt-get install te-va-unlock -y – jailbreak the TE agent to allow additional sudo access
apt-get install net-tools ifmetric wireless-tools -y – install tools to allow wireless functionality
ifconfig wlan0 up – bring up a wireless LAN interface
wpa_passphrase <SSID> <SSID password> > /etc/wpa_supplicant.conf – configure SSID and PSK values for the wireless connection and write them to the wpa_supplicant.conf file
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf -D wext – imports the wpa_supplicant file and configures the wireless LAN interface to use the wext driver
ifmetric eth0 200 – set the wired interface to have a lower priority than the wireless interface
sed -i ‘1iauto wlan0niface wlan0 inet dhcp’ /etc/network/interfaces – configure DHCP for the wireless interface
dhclient wlan0 – enable DHCP for the wireless interface
systemctl mask apt-news.service – prevent apt-news.service from running. See the Wireless TE Agent Update Failure section for more information
systemctl mask esm-cache.service – prevent the esm-cache.service from running. See the Wireless TE Agent Update Failure section for more information

Wireless TE Agent Update Failure

During testing of the wireless TE agent for Raspberry Pi, it was noticed that the agents lost connectivity with the ThousandEyes web manager after a variable amount of time. Troubleshooting multiple failed agents identified logs associated with updates to the apt-news.service and esm-cache.service processes immediately before failure of the wlan:

syslog

Aug  6 02:17:04 registration-wireless systemd[1]: Started PackageKit Daemon.
Aug  6 02:17:20 registration-wireless systemd[1]: Starting Update APT News…
Aug  6 02:17:20 registration-wireless systemd[1]: Starting Update the local ESM caches…
Aug  6 02:17:21 registration-wireless systemd[1]: apt-news.service: Succeeded.
Aug  6 02:17:21 registration-wireless systemd[1]: Finished Update APT News.Aug  6 02:17:22 registration-wireless systemd[1]: esm-cache.service: Succeeded.
Aug  6 02:17:22 registration-wireless systemd[1]: Finished Update the local ESM caches.Aug  6 02:18:38 registration-wireless systemd-networkd[667]: wlan0: Lost carrier

More information on the two services can be found here:

https://askubuntu.com/questions/1452519/what-are-the-services-apt-news-and-esm-cache-and-how-do-i-disable-them

Masking the services so they are not run by systemd resolved the failure of wlan0:

$ sudo systemctl mask apt-news.service
$ sudo systemctl mask esm-cache.service

The above commands were then added to the startup scripts.

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

InstagramFacebookTwitterLinkedIn

  Read how Validated Design TMEs deployed ThousandEyes on Pi4s wirelessly on the Black Hat USA 2023 network to provide network assurance and actionable insights.  Read More Cisco Blogs 

How do we achieve success in this industry? It was the question I pondered 20 years ago when I set out to take my career to the next level and attain what few have: the Cisco Certified Internetwork Expert (CCIE) certification.<span data-ccp-props=' Facebook '> 

Now leading the CCIE program as Cisco’s Director of Certifications, I’ve experienced firsthand the revolutionary impact of this program on my peers and the industry.  

This year, we celebrate the 30-year anniversary of the CCIE certification. Let’s look back at where we came from—and how far we’ve come. 

Brief History of the CCIE Program 

1990: Networkers Users Symposiums launches: As networking booms with new levels of connection and more challenges in connectivity, we launch Networkers Users Symposiums – also known as “Networkers.” The conference, later renamed Cisco Live, enables customers to learn, collaborate, and grow the industry together.  
1993: CCIE launch: Cisco announced the CCIE program publicly on September 27, 1993, when former CEO John Chambers was quoted, “The CCIE program begins where other vendors’ certification programs leave off.”  
Fun fact: The original name of the program was going to be “Cisco Top Gun.” However, the famous Tom Cruise movie by the same name had already been released. We figured there may be some legal issues in using that name… so instead, we chose “Cisco Certified Internetwork Expert.”
1993: First CCIE Award: The CCIE program chose to start its numbering system with 1024 (2 to the power of 10). That number went into a plaque on the CCIE lab itself in San Jose, CA, followed by number 1025, awarded to the first person to earn the CCIE. And the rest is history.  

My CCIE Journey 

Inside the Sydney lab with my daughter, 20 years apart.

My interest in becoming a CCIE started in the late 90s. Tech was very different from today. The internet was evolving, and networking was a completely new concept. I was looking for a challenge to advance my career.  

When researching how to take my career to the next level, I discovered Cisco, a pretty new tech company at the time. Not only did I find a way to test my skills with the CCIE program, but a window of opportunity. The exam ensured perfection, and I liked the challenge.  

Becoming a CCIE opened career advancements for me. Not only was I able to receive promotions with more pay, but I also gained more peer respect and recognition—as did my colleagues who received the certification. There is a high level of respect for people who achieve CCIE. You become part of an elite group.

In general, value is directly equivalent to the relevancy of a product. If the CCIE test was irrelevant to what I did in my job, I would not have pursued it. And if it was not updated to the technology of today, the program would fail. I’ve been a CCIE for 20+ years and can still apply my learnings from the program. 

CCIE program is a true test of deep knowledge and experience, an endorsement (validation) of being an expert – crème de la crème! 

The CCIE Program’s Transformation Over Time 

As technology has evolved, the program has changed significantly. In the early days, the CCIE program was a two-day test, where candidates spent the first day setting up the hardware and initializing the pod with basic configurations. Then, candidates would come back for a second day to perform more advanced configurations, spending the second half of the day troubleshooting scenarios. 

Today’s CCIE program has been streamlined to a one-day test, removing the now trivial steps and focusing more on network strategy, architecture, and the network lifecycle, including planning and designing to meet customer requirements. The enhanced lab exam is focused on the end-to-end network lifecycle from Day 0 (planning & designing), Day 1 (implementing and deploying), and Day 2 (operating & maintaining), testing comprehensive skills across the entire spectrum of network solutions. 

Starting last year, we introduced Cisco Certification Roadmaps, an agile and iterative cadence-based model to update exams more frequently and incrementally, keeping them more current and relevant, aligned with rapid technology evolution, and ensuring relevancy for today. The roadmap will provide timely and relevant information to keep you ahead of changes being introduced as you study for the CCIE exam and all other Cisco certifications.  

Watch Snack Minute Episode 107 to learn more about Cisco certification updates, understanding major versus minor revisions, and staying in the know. Watch now.

Join the CCIE Celebration

Join us in celebrating the 30th anniversary of the program and become a part of the #CCIE30 community! 

In June, we kicked off celebrations at Cisco Live in Las Vegas, but we will be celebrating around the world throughout the year. Each CCIE lab site will host “birthday” celebrations, bringing customers and employees together to share stories and discuss how the CCIE program changed their lives.  

How Can You Become a CCIE? 

This exam is about validating your experience—in the field and in real life. 

Most people who fail lack experience. I would know. I failed previously, not due to lack of studying but because I was still too new in my career. Once I started actually doing the scenarios presented in the exam in my job, my learning accelerated, and so did my CCIE prep. It was all about getting hands-on and practicing. 

Even once you have ideally gained five to seven years of experience, you still must prepare. Becoming a CCIE is hard work, but we have resources available to you.  

Whether you are on your journey to becoming a CCIE or working toward advancing your career in general, I leave you with this advice: Focus on pursuing excellence, and success will follow. 

 

Join the CCIE Community on the Cisco Learning Network.

Some will say that DevNet started at Cisco Live! in May of 2014 when the event itself was celebrating its 25th anniversary. The 2nd floor of Moscone Center South provided the space for the very first DevNet Zone where the ragtag group of DevNet staff found themselves immersed in a sea of automation neophytes from the minute the doors opened. After scrambling for two months to prepare and a sleepless weekend hosting the very first DevNet hackathon, promoting Cisco technologies like MSE, Jabber, and Cisco Prime, the already exhausted staff was energized to find so many people excited to learn about APIs and coding. The energy was high, the vibe was palpable, and John Chambers was in the room. DevNet had arrived.

In the beginning…

Some may also say that DevNet started in December of 2013, when, after months of an intense effort by an inspired group of doc writers, developers, QA, DBAs, systems architects, and BDMs, developer.cisco.com launched. Introducing the world to Cisco docs that weren’t PDFs and a clever innovation called the Sandbox. DevNet was Cisco’s foray into the software world from its hardware roots. Now, those ISVs and SIs who wanted to build solutions could test out Cisco’s programmability opportunities without building their own labs and spending tens to hundreds of thousands of dollars.  

But the truth is DevNet started 10 years ago, this week, in a handful of meetings held in Round Rock, TX, and I had the fortune of being there. The precursor to DevNet was called Cisco Developer Network (CDN) and prior to that it was Cisco Technology Developer Program (CTDP). Both of these programs were successful in their own right, but the focus was always only on partners. DevNet was intended to be a “startup inside of a company” that provided a true enablement program for all developers. The primary goal was to remove the barriers to entry that lay in front of those wanting to build solutions on top of Cisco platforms. At that tête-à-tête, the foundations for DevNet were laid, and have guided the program ever since.

The first DevNet Zone, 2014 Cisco Live US in San Francisco, revealed overwhelming interest in Cisco’s software direction

The “hair on fire” years

The first few years felt like we were flying around with our hair on fire. Some things were fantastic, like the initial launch of Learning Labs, our partnership with Networking Academy, and the DevNet Zone, a staple now at Cisco Live!. Other things we stumbled on, like week after week of hackathons where students barely understood Cisco technologies let alone wanted to build something with them. But whether it was a success or a failure, we always were able to focus on the key to everything; helping people with code building and API usage to make their jobs, solutions, and organizations better.

As our experience grew, we knew we needed to really build our community to scale the message, and that’s where the real fun began. First off, we launched DevNet Express, an “in-the-box” program that allowed our Sales System Engineers to take DevNet’s programmability content and deliver it, with interactive labs, directly to their customer base. Over three years, this program became so popular that we were holding or supporting 300+ events per year, including pre-Cisco Live! workshops hosting 100+ developers at a time.

Following the success of that effort came DevNet CREATE, our standalone event started in 2017. The impetus for CREATE was that it be “more than a Cisco conference.” And that it was. Hundreds of developers came to DevNet CREATE to pow-wow on topics relating to DevOps, SecOps, ML, philanthropic technology, and general software development. Not to mention the event-within-an-event Camp CREATE where 30 selected developers were split into innovation teams to build cutting-edge solutions. Let’s just say you haven’t lived until you’ve traversed and diagnosed your network in VR. It was unlike anything Cisco had ever seen!

A community that knows how to work, innovate, and have fun doing it!

That word, “community,” really doesn’t capture the full effect of the people that contributed to it. Year after year, event after event, we benefitted from partners, SEs, learners, customers, and internal engineers that supported and spread the word of DevNet. Network engineers who had been doing their job for multiple decades were now seeing their colleagues writing code to build solutions and make their job more interesting and wanted in on the fun. Students and earlier in career technologists saw the value in adding automation understanding to their operations education. At DevNet CREATE we would attempt, as best we could, to recognize those who really acted as an extra advocate for DevNet by presenting them with our Creator Awards. This recognition showed that we were not just paying attention, but really appreciated our community.

The DevNet Community celebrates becoming 500,000 members strong at the
“DevNet Social Club” event at Cisco Live in Orlando, Florida

But it’s not all work and no play. DevNet knew how to have fun. The capstone for our community social life was the DevNet Social Club; an inclusive party for all our fellow DevNet-ers at Cisco Live! Flamenco dancers and rock cellists ushered in the good times as we celebrated with everyone in our community.

DevNet gets professional

Piggybacking on that success, we found that our efforts in education, especially with Cisco Learning Network and Networking Academy, needed validation. That’s where DevNet’s certification program came in. Over the course of two years, while doing ALL of the above, our diligent team worked with our colleagues in Cisco Learning and Certifications to build exams and course material that could be worthy of the brand of a Cisco Certification. If you’ve never written exam questions for multiple levels of technical exams, I’d invite you to try it sometime. It’s a maddening foray into being VERY picky about wording and Bloom’s hierarchy of learning. We made it through though, and all that effort culminated in Cisco’s first launch of a programmability certification track in February of 2020. 

Then, as we all know, the world was hit with calamity. But, through it, DevNet adjusted. Virtual events became a necessity as we held DevNet CREATE and the DevNet Zone at Cisco Live! online for thousands of attendees. We were able to facilitate virtual exams for those who wanted to make sure they used the lock-down time to bolster their resume. It even gave us time to work on some ideas we had swirling in the back of our heads but never had time to focus on, like the Snack Minute video series.  

More to come

Fortunately, we’ve been able to return to live events holding our first DevNet Zone in person at Cisco Live! 2022.  It was a more reserved event with safety protocols in place, but the audience for programmability and automation content was just as hungry as ever. The week proved successful with full classrooms and workshops. We saw an even bigger response for the events that followed in 2023 in Amsterdam and Las Vegas.  

The DevNet staff poses in the DevNet Zone at Cisco Live US 2023

Through it all, I was fortunate to see DevNet and its people grow, change, pivot, and in some instances, survive. It’s been the best and most interesting challenge of my career. I always tell anyone who asks (and some of those who don’t) that I love my job and I’m never bored. This is because I believe in what we do here and the people we help in our community.  There are only a handful of us from the original group remaining in the team, but the influence of all of the hundreds of people that made DevNet what it is across 10 years cannot be overstated.  As we look to the next 10 years, I am extremely excited about DevNet’s next chapter, but it’s always fun to look back and see where we’ve come from. 

See what’s new in DevNet today.

​

Some will say that DevNet started at Cisco Live! in May of 2014 when the event itself was celebrating its 25th anniversary. The 2nd floor of Moscone Center South provided the space for the very first DevNet Zone where the ragtag group of DevNet staff found themselves immersed in a sea of automation neophytes from the minute the doors opened. After scrambling for two months to prepare and a sleepless weekend hosting the very first DevNet hackathon, promoting Cisco technologies like MSE, Jabber, and Cisco Prime, the already exhausted staff was energized to find so many people excited to learn about APIs and coding. The energy was high, the vibe was palpable, and John Chambers was in the room. DevNet had arrived.

In the beginning…

Some may also say that DevNet started in December of 2013, when, after months of an intense effort by an inspired group of doc writers, developers, QA, DBAs, systems architects, and BDMs, developer.cisco.com launched. Introducing the world to Cisco docs that weren’t PDFs and a clever innovation called the Sandbox. DevNet was Cisco’s foray into the software world from its hardware roots. Now, those ISVs and SIs who wanted to build solutions could test out Cisco’s programmability opportunities without building their own labs and spending tens to hundreds of thousands of dollars.  

But the truth is DevNet started 10 years ago, this week, in a handful of meetings held in Round Rock, TX, and I had the fortune of being there. The precursor to DevNet was called Cisco Developer Network (CDN) and prior to that it was Cisco Technology Developer Program (CTDP). Both of these programs were successful in their own right, but the focus was always only on partners. DevNet was intended to be a “startup inside of a company” that provided a true enablement program for all developers. The primary goal was to remove the barriers to entry that lay in front of those wanting to build solutions on top of Cisco platforms. At that tête-à-tête, the foundations for DevNet were laid, and have guided the program ever since.

The first DevNet Zone, 2014 Cisco Live US in San Francisco, revealed overwhelming interest in Cisco’s software direction

The “hair on fire” years

The first few years felt like we were flying around with our hair on fire. Some things were fantastic, like the initial launch of Learning Labs, our partnership with Networking Academy, and the DevNet Zone, a staple now at Cisco Live!. Other things we stumbled on, like week after week of hackathons where students barely understood Cisco technologies let alone wanted to build something with them. But whether it was a success or a failure, we always were able to focus on the key to everything; helping people with code building and API usage to make their jobs, solutions, and organizations better.

As our experience grew, we knew we needed to really build our community to scale the message, and that’s where the real fun began. First off, we launched DevNet Express, an “in-the-box” program that allowed our Sales System Engineers to take DevNet’s programmability content and deliver it, with interactive labs, directly to their customer base. Over three years, this program became so popular that we were holding or supporting 300+ events per year, including pre-Cisco Live! workshops hosting 100+ developers at a time.

Following the success of that effort came DevNet CREATE, our standalone event started in 2017. The impetus for CREATE was that it be “more than a Cisco conference.” And that it was. Hundreds of developers came to DevNet CREATE to pow-wow on topics relating to DevOps, SecOps, ML, philanthropic technology, and general software development. Not to mention the event-within-an-event Camp CREATE where 30 selected developers were split into innovation teams to build cutting-edge solutions. Let’s just say you haven’t lived until you’ve traversed and diagnosed your network in VR. It was unlike anything Cisco had ever seen!

A community that knows how to work, innovate, and have fun doing it!

That word, “community,” really doesn’t capture the full effect of the people that contributed to it. Year after year, event after event, we benefitted from partners, SEs, learners, customers, and internal engineers that supported and spread the word of DevNet. Network engineers who had been doing their job for multiple decades were now seeing their colleagues writing code to build solutions and make their job more interesting and wanted in on the fun. Students and earlier in career technologists saw the value in adding automation understanding to their operations education. At DevNet CREATE we would attempt, as best we could, to recognize those who really acted as an extra advocate for DevNet by presenting them with our Creator Awards. This recognition showed that we were not just paying attention, but really appreciated our community.

The DevNet Community celebrates becoming 500,000 members strong at the
“DevNet Social Club” event at Cisco Live in Orlando, Florida

But it’s not all work and no play. DevNet knew how to have fun. The capstone for our community social life was the DevNet Social Club; an inclusive party for all our fellow DevNet-ers at Cisco Live! Flamenco dancers and rock cellists ushered in the good times as we celebrated with everyone in our community.

DevNet gets professional

Piggybacking on that success, we found that our efforts in education, especially with Cisco Learning Network and Networking Academy, needed validation. That’s where DevNet’s certification program came in. Over the course of two years, while doing ALL of the above, our diligent team worked with our colleagues in Cisco Learning and Certifications to build exams and course material that could be worthy of the brand of a Cisco Certification. If you’ve never written exam questions for multiple levels of technical exams, I’d invite you to try it sometime. It’s a maddening foray into being VERY picky about wording and Bloom’s hierarchy of learning. We made it through though, and all that effort culminated in Cisco’s first launch of a programmability certification track in February of 2020. 

Then, as we all know, the world was hit with calamity. But, through it, DevNet adjusted. Virtual events became a necessity as we held DevNet CREATE and the DevNet Zone at Cisco Live! online for thousands of attendees. We were able to facilitate virtual exams for those who wanted to make sure they used the lock-down time to bolster their resume. It even gave us time to work on some ideas we had swirling in the back of our heads but never had time to focus on, like the Snack Minute video series.  

More to come

Fortunately, we’ve been able to return to live events holding our first DevNet Zone in person at Cisco Live! 2022.  It was a more reserved event with safety protocols in place, but the audience for programmability and automation content was just as hungry as ever. The week proved successful with full classrooms and workshops. We saw an even bigger response for the events that followed in 2023 in Amsterdam and Las Vegas.  

The DevNet staff poses in the DevNet Zone at Cisco Live US 2023

Through it all, I was fortunate to see DevNet and its people grow, change, pivot, and in some instances, survive. It’s been the best and most interesting challenge of my career. I always tell anyone who asks (and some of those who don’t) that I love my job and I’m never bored. This is because I believe in what we do here and the people we help in our community.  There are only a handful of us from the original group remaining in the team, but the influence of all of the hundreds of people that made DevNet what it is across 10 years cannot be overstated.  As we look to the next 10 years, I am extremely excited about DevNet’s next chapter, but it’s always fun to look back and see where we’ve come from. 

See what’s new in DevNet today.

  As DevNet turns 10, we're excited about DevNet’s next chapter. But, it’s always fun to look back and see where we’ve come from.  Read More Cisco Blogs 

In our March 2023 blog, “What is EPSS and Why Does It Matter?”, Michael Roytman, Distinguished Engineer at Cisco (former Chief Data Scientist at Kenna Security) and co-creator of EPSS, covers the rol… Read more on Cisco Blogs

​

In our March 2023 blog, “What is EPSS and Why Does It Matter?”, Michael Roytman, Distinguished Engineer at Cisco (former Chief Data Scientist at Kenna Security) and co-creator of EPSS, covers the role the Exploit Prediction Scoring System (EPSS) plays in a security program. To sum it up, EPSS enables practitioners to have a defensible way to forecast how likely a newly published vulnerability is to become exploited before attackers have a chance to build new ransomware or exploits.

In this blog, we’ll cover more details about EPSS, how it compares to CVSS, as well as the role it plays in Cisco Vulnerability Management’s risk scoring.

Digging Deeper: The Importance of EPSS

EPSS is an open-source, “data-driven effort for estimating the likelihood (probability) that a software vulnerability will be exploited in the wild” (FIRST.org). Its overall goal is to help security teams better prioritize vulnerability remediation work.

Fun fact: Cisco (formerly Kenna Security) licenses the patent “Exploit Prediction Based on Machine Learning” to FIRST.org to enable EPSS development.

Anonymized data from the Cisco Vulnerability Management platform was used by the creators of EPSS to compare which vulnerabilities were being exploited in the wild to which vulnerabilities organizations were remediating. The findings revealed that remediation strategies were inconsistent and ad-hoc. Based on the evidence collected that showed what was being exploited, the creators built a data model to predict exploitability.

EPSS vs CVSS: What’s the Difference?

EPSS was initially inspired by the Common Vulnerability Scoring System (CVSS). CVSS assigns scores to vulnerabilities based on their principal characteristics; the score indicates the severity of a vulnerability, providing a range from 0.0 to 10.0 (the higher the score, the greater severity). CVSS can be categorized into low, medium, and high severity, and organizations can use CVSS to help prioritize vulnerabilities that exist in the system. However, CVSS on its own doesn’t indicate a likelihood of exploitation, leading to criticisms that call out its ineffectiveness in prioritizing and predicting threats.

EPSS, on the other hand, estimates the probability that a vulnerability will be exploited in the wild in the next 30 days, with a score ranging between 0 to 1. EPSS looks at two key prioritization strategies: coverage and efficiency. Coverage is the proportion of vulnerabilities with known exploitation activity that are prioritized. Efficiency is the proportion of all prioritized vulnerabilities with known exploitation activity. Despite its ability to help in predicting which vulnerabilities will be exploited in the wild, EPSS doesn’t provide all the information needed to deprioritize vulnerabilities, which makes it difficult to make decisions on what to fix first.

Coupling EPSS and CVSS scoring data enables organizations to more effectively prioritize vulnerabilities based on both severity and probability of exploitation. Even so, there are other data sources like real-time threat data that should be incorporated into vulnerability prioritization scoring for optimized results. More on that in just a bit.

What It Means for Cisco Vulnerability Management Customers

Risk Scoring in the Cisco Vulnerability Management platform helps customers prioritize the vulnerabilities that pose the greatest risk to their specific organizations, while deprioritizing the ones that don’t. Our risk score is continuously evolving to include the latest inputs for the most accurate prioritization. This update easily enables customers to identify and remediate top priority vulnerabilities based on the prediction that it will become an Active Internet Breach in the near future.

Figure 1: Explore page in Cisco Vulnerability Management platform

While it’s important to understand a vulnerability may be exploited in the future, it’s even more important to know which vulnerabilities are already being exploited. That’s why, in conjunction with EPSS and CVSS, Cisco Vulnerability Management risk scoring incorporates an organization’s internal security data and threat and exploit intelligence from 19+ feeds, including Cisco Talos, to not only determine how risky a vulnerability is, but to also understand the volume and velocity at which the vulnerability is being targeted. By leveraging the risk score in Cisco Vulnerability Management, customers can determine which vulnerabilities pose the biggest risk to their organization and which vulnerabilities are low risk and, therefore, can be deprioritized. The result is that customers are focusing their limited resources on remediating the vulnerabilities that matter most.

In addition to identifying which vulnerabilities are most likely to result in an exploit, Cisco Vulnerability Management uses Risk Meter scoring to also highlight the impact of those exploits by measuring the risks of assets, groups of assets, and organizations. With accurate and quantifiable risk scores, customers can understand their organizations’ current risk posture and identify the actions needed to reduce the greatest amount of risk.

Interested in learning more about EPSS? Check out the site and browse the data (it’s open and free): www.first.org/epss

Want to take a deeper look at Cisco Vulnerability Management? Visit our page: https://www.cisco.com/site/us/en/products/security/vulnerability-management/index.html

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

InstagramFacebookTwitterLinkedIn

  Learn about EPSS, how it compares to CVSS, and the role it plays in Cisco Vulnerability Management’s risk scoring.  Read More Cisco Blogs