In today’s fast-paced and hyper-connected world, gone are the days when deploying network devices required sending an expert to each location — a cumbersome, time-consuming, and error-prone process… Read more on Cisco Blogs

​

In today’s fast-paced and hyper-connected world, gone are the days when deploying network devices required sending an expert to each location — a cumbersome, time-consuming, and error-prone process that caused significant downtime and increased operational costs. To surmount these barriers, Cisco offers a variety of network orchestrators. These included Cisco Catalyst Center (formerly Cisco DNA Center), SD WAN Manager (formerly Cisco vManage), and Meraki Dashboard, which assist businesses in automating their campus network management including Day 0 provisioning. These orchestrators allow network administrators to remotely deploy a large number of network devices quickly and securely, without requiring any human intervention. This not only saves time and money but also liberates IT department resources, allowing them to redirect their efforts towards other critical areas.

Using Catalyst Center PnP, Cisco IT was able to reduce annual deployment costs for some sites by approximately 25%, or more than $1.6 million. Additionally, upgrading our 285 small and medium-sized offices with Cisco Catalyst Center saved 570 man-hours per upgrade[1].

In addition to Cisco network orchestrators for customers employing a Do-It-Yourself (DIY) approach with homegrown tools, Catalyst 9000 series switches offer support for an assortment of open standard-based implementations for Day 0 network automation, such as Preboot eXecution Environment (PXE) and Zero Touch Provisioning (ZTP). So, if you find yourself still manually configuring network devices, it may be time to consider stepping out of the stone age and exploring the benefits of automation.

Day 0 network automation

When delving into the realm of open standard-based Day 0 network automation, it becomes clear that PXE, while a useful technique, comes with a set of limitations, such as only allowing network devices to boot from a network-based source and not being able to send configurations to devices during the PXE workflow. ZTP, on the other hand, can be used to upgrade software images and push configuration files, reducing the chance of human error and ensuring configuration consistency in order to get network devices up and running.

While ZTP and PXE are convenient for automating the provisioning process, they may inadvertently expose network devices to potential threats. Lack of secure authentication and verification mechanisms during the provisioning process is one of the primary concerns with these techniques. Furthermore, ZTP and PXE utilize HTTP/TFTP to download the software image or configuration files, which are inherently insecure protocols because they lack encryption. Due to these limitations, these techniques could result in unauthorized access to the device or a man-in-the-middle attack if the right security measures aren’t put in place during the device provisioning.

Cyberattacks have increased

In today’s rapidly evolving digital landscape, where enterprises are undergoing substantial transformation, cyberattacks have increased amid the rise of cloud computing, hybrid and multi-cloud networks, and the rise of remote work. According to the latest IBM Ponemon Institute 2023 Cost of Data Breach Study, the average cost of a data breach reached an all-time high in 2023 of USD 4.45 million [2]. Furthermore, according to ITIC’s 2022 Global Server Hardware Security report, 76% of firms cite Data Breaches and Human Error as the leading reason of server, OS, application, and network downtime, and the hourly cost of downtime has risen to over $300,000[3].

Given that cybercriminals are constantly devising new techniques to infiltrate networks, the traditional security approach, which assumes that everything within the network perimeter is trustworthy, is no longer sufficient. This is also true for Day 0 network automation, where it is crucial to validate the trustworthiness of the newly deployed device, bootstrap server, and configurations pushed to the device. Without implementing these security measures, our networks are vulnerable to a variety of cyberattacks, including the notorious zero-day exploits. To ensure maximal security and minimize potential risks, the Zero Trust principle of “never trust, always verify” must be implemented throughout the entire provisioning process.

Maintain security throughout the provisioning process

This is where Secure Zero Touch Provisioning comes into play. Secure ZTP, as described in RFC 8572, is an enhanced version of ZTP that emphasizes maintaining security throughout the provisioning process by reducing the likelihood of security breaches. Secure ZTP is a proactive approach that employs robust authentication, a secure boot mechanism, and encrypted communication channels to enhance the security posture of a network while Day 0 network automation is in place.

How does Secure ZTP work?

Secure ZTP employs three-step validation, including device validation, server validation, and artifact validation, to securely onboard the device. The diagram provided below illustrates the various steps involved in the device onboarding and provisioning process within a secure ZTP framework. Let’s take a closer look at each of these steps:

1. Device Validation

Before onboarding a new device on the network, it is crucial to ensure that neither the device nor its firmware has been tampered with or compromised to prevent supply chain or any other attacks, in which malicious actors attempt to introduce modified or malicious devices into the network. Based on the recent IBM report, 15% of organizations identified a supply chain compromise as the source of a data breach [2].Secure ZTP performs device authentication prior to provisioning it in order to verify the integrity and authenticity of a device and to allow only authorized devices to join the network.For device validation, Secure ZTP uses certificate-based authentication where the device sends the Trust Anchor Certificate (also known as a SUDI certificate installed in the device during the manufacturing process) to the Secure ZTP server, and the server validates it with the public certificate (provided by the manufacturer) to ensure the device’s authenticity.

2. Server Validation

Server validation is another vital part of the Secure ZTP. By confirming the server’s identity, the device can assure that it is communicating with an uncompromised, trustworthy server. This prevents unauthorized or malicious servers from intercepting or manipulating the provisioning process. After verifying the device, bootstrap server sends server certificate. The device requests bootstrapping data with the flag “signed-data-preferred” after receiving the server certificate, indicating that the device does not trust the server. In this case, keep in mind that server validation is optional in Secure ZTP. If the network administrator decides to perform server validation (which entitles server to receive bootstrapping progress report), the server will send the “redirect-data” with other bootstrapping data to the device, providing its own address and the trust anchor. The device verifies the server’s certificate and marks it as trusted server after receiving the trust anchor. Here, if the system administrator opts not to validate the server, the server will instead pass on bootstrapping data in place of the “redirect-data”. In addition, the device will continue the bootstrapping process assuming the server is untrusted.

3. Artifact Validation

Artifact validation is important to ensure that the configuration files or software images used to provision network devices are authentic and have not been tampered with. Once the server validation is complete (or skipped), the bootstrap server will send the owner certificate, ownership voucher, and onboarding information to the device as bootstrapping data. Let’s discuss them closely to gain a better understanding.

Ownership Voucher (OV): The ownership voucher artifact validates the owner certificate to verify the identity of the device’s owner. The device manufacture signs the OV and provides it to the customer based on the request. To generate the OV, the customer must provide the pinned-domain-cert and serial number of the device to the Cisco MASA server.
Owner Certificate (OC): Owner Certificate is an X.509 certificate that binds an owner identity to a public key, which a device can use to validate signature over the conveyed information artifact. The owner certificate also holds all intermediate certificates that led to the “pinned-domain-cert” certificate specified in the ownership voucher, allowing the OV to validate the OC.
Conveyed Information/Onboarding Information: Onboarding information provides data necessary for a device to bootstrap itself and establish secure connections with other systems. Onboarding information specify details about the boot image a device must be running, an initial configuration the device must commit, and scripts that the device must successfully execute. The onboarding information must be signed by the device’s owner using OC.

Zero Trust is crucial when performing Day 0 provisioning

In addition to its many features, Secure ZTP goes beyond by offering audit trails and monitoring capabilities. This includes logging all provisioning events, configuration changes, and user actions. By monitoring ZTP activities, network administrators can quickly detect any suspicious activity and take appropriate action.

As we wrap up our discussion, it becomes clear that Zero Trust is also crucial when performing Day 0 provisioning, and Secure ZTP is the best way to ensure that zero trust principles are applied while performing Day 0 provisioning using a Do-It-Yourself (DIY) approach.

With the IOS-XE 17.11.1 release, users can now take advantage of the secure Zero Touch Provisioning (ZTP) capabilities with Catalyst 9000 series switches. This exciting feature aligns with the specifications outlined in RFC 8572, ensuring a secure and seamless provisioning experience. For more details about how to implement Secure ZTP, please refer the IOS-XE 17.11.1 Configuration Guide.

Keep Learning with these resources

Visit the Zero Touch Resource Hub
Learn more about Cisco IOS XE
Cisco IOS XE blogs
Zero Touch Provisioning blog

References

Cisco DNA Center: Early Results from Intent-based Networking
Security, Data Breaches Top Cause of Downtime in 2022
IBM – Cost of a Data Breach Report 2023

  Learn how Cisco Secure ZTP (Zero Touch Provisioning) helps you maintain security throughout the provisioning process with a proactive approach that employs robust authentication and encrypted communication channels to enhance the security posture of a network while Day 0 network automation is in place.  Read More Cisco Blogs 

As organizations build, modernize, and deploy business-critical applications including traditional and on-premise, hybrid applications, and modern, distributed (cloud native), they need more… Read more on Cisco Blogs

​

As organizations build, modernize, and deploy business-critical applications including traditional and on-premise, hybrid applications, and modern, distributed (cloud native), they need more observability into their increasingly complex, dispersed workloads. Using multiple domain-specific monitoring, visibility, and observability tools can provide insights into individual domains, but it’s difficult to do at scale, and creates multiple sources of truth. How can organizations bring together enterprise workload details to the insights they need for better application experiences?

Breaking through layers of complexity with full stack observability

Cisco FSO Platform is an open and extensible, API-driven platform focused on OpenTelemetry and anchored on metrics, events, logs, and traces (MELT), providing AI/ML driven analytics. It brings data together from multiple domains including application, networking, infrastructure, security, cloud, sustainability, and business sources.

Cisco FSO platform also empowers a new observability ecosystem delivering relevant and impactful business insights through new use-cases and extensions. Cisc  flexible modules that help customers with specific business outcomes. One great example is the vSphere Observability and Data Modernization built by Cisco technology partner CloudFabrix.

CloudFabrix vSphere Observability and Data Modernization

Many organizations are utilizing the VMware vSphere enterprise workload platform for their business-critical applications. So CloudFabrix took the initiative to develop a module that helps them observe vSphere through Cisco FSO Platform for a more centralized, consolidated view.

“Most of our customers today are running hybrid environments, so we decided our first module should bring vSphere into Cisco FSO Platform, to make them “first class citizens” of FSO,” said Raja Penmetsa, CTO of CloudFabrix. “CloudFabrix collects the infastructure information underneath your application, translates into OTEL telemetry signals, and puts it into FSO.”

CloudFabrix enables organizations to acquire a vast amount of data and entity insights from across the technology stack make them easily accessible on Cisco FSO Platform, to support application or infrastructure modernization, performance impact analysis, and other use cases.

The module also enriches vSphere and VMware vRealize Operations data with their environment’s Kubernetes and infrastructure data.

CloudFabrix provides visibility and correlation into where services are running and how they are connected, including details about which Kubernetes service is running on a particular VMware host, on a specific VM.

Our technology partners like CloudFabrix say that Cisco FSO Platform has enabled them to hit the ground running as they commence work on expansion modules. Its rich modeling available right out of the box helps them build and bring new solutions to market fast, and rapidly fill use cases for specific domains and vertical use cases.

Seizing emerging market opportunities and use cases faster

Our technology partners like CloudFabrix say that Cisco FSO Platform has enabled them to hit the ground running as they commence work on expansion modules. Its rich modeling available right out of the box helps them build and bring new solutions to market fast, and rapidly fill use cases for specific domains and vertical use cases.

“First and foremost, Cisco FSO Platform has done a really good job of enabling us to extend the platform,” said Penmetsa. “Based on our past experiences, something like this would have taken about six to 12 months to build. Working with the Cisco FSO Platform for the first time, it took us less than four weeks to build our vSphere Observability and Data Modernization module. And now that we’re familiar with the platform, we should be able to churn out future deliverables on our roadmap even faster.”

Watch the full interview between Raju Penmetsa, CTO, Cloudfabrix and Sunder Parameswaran, VP Product Management, Cisco FSO.

As Cisco continues to enhance the platform with additional tools and documentation for developers, CloudFabrix and other technology partners will gain access to even more robust support.

Together, Cisco FSO Platform and its partner modules deliver a comprehensive solution that scales with the pace of business, with easy extensibility everywhere, across the infrastructure and the application life cycle.

Learn more about  Cisco FSO Platform and the CloudFabrix Module

  If you utilize VMware vSphere enterprise workload platform for your business-critical applications, take a look at the CloudFabrix module that helps you observe vSphere through Cisco FSO Platform for a more centralized, consolidated view.  Read More Cisco Blogs 

After the executive order to bolster the nations cybersecurity following the Colonial Pipeline attack, the U.S. Transportation Security Administration (TSA) has been releasing new mandates for… Read more on Cisco Blogs

​

After the executive order to bolster the nations cybersecurity following the Colonial Pipeline attack, the U.S. Transportation Security Administration (TSA) has been releasing new mandates for critical infrastructure such as freight and passenger rail, pipelines, and airports, with more industries to follow.

The networks that support these critical infrastructures are mission-critical, which means that it is essential to be able to stay connected while securely administering policy in the industrial space. Being an industry leader in networking and security across both the information technology (IT) and operational technology (OT) domains, Cisco is in a unique position to deliver an end-to-end security strategy, while enhancing operational uptime and resiliency.

To strengthen the cybersecurity posture of the nation’s critical infrastructure, there are four key requirements outlined by the mandates, highlighted in bold text below.

Network segmentation

The first requirement is to “Implement network segmentation policies and controls to ensure that the Operational Technology (OT) system can continue to safely operate if an Information Technology (IT) system has been compromised.”

Using a defense-in-depth approach, Cisco addresses this requirement in many parts of the network, adapting to the unique architecture needs of individual organizations. The solution is a common one, use the network infrastructure to segment a network. Do not wait until you reach a “security appliance” to do security. Cisco provides an end-to-end segmentation solution in which data is kept within its own virtual network from source to destination, wherever that may be.

To support the unique requirements of industrial networks, the reach of Cisco SD-WAN has been expanded through Cisco Industrial Routers, which provide the connectivity, mobility, and security required for critical infrastructure. SD-WAN segments traffic at the edge of the network and maintains separation through all relevant points in the network. Policy can be orchestrated across multiple enforcement points in the network using Cisco Catalyst SD-WAN, or—if your organization prefers—can support the evolution to a secure service edge (SSE) with Cisco Secure Access.

Access control

TSA highlights the need to “Implement access control measures to secure and prevent unauthorized access to Critical Cyber Systems.” As OT devices traverse both the LAN and the WAN with a unified identity, Cisco can enforce policy everywhere. Cisco Security Group Tags (SGTs) identify the role that a device has on the network, and the associated privileges are enforced by switches, routers, and firewalls, depending on where the data flows.

Remote users, whether internal technicians or vendor support, often need access to critical cyber systems. Cisco Secure Equipment Access (SEA) provides flexible access for remote configuration and maintenance of industrial assets in distributed locations while minimizing security risk.

Continuous monitoring

Segmentation is not enough to complete a security solution. By implementing “continuous monitoring and detection policies and procedures to detect cybersecurity threats and correct anomalies that affect Critical Cyber System operations,” we can continually monitor and evaluate the trust of both users and devices on our networks and push policy back into the network as security posture changes.

To provide visibility and security posture to the industrial network, Cisco Cyber Vision is embedded in Cisco networking infrastructure in order to avoid the need for dedicated appliances and/or costly Switched Port Analyzer (SPAN) solutions. Cyber Vision identifies assets, their characteristics, and their communication patterns to “reduce the risk of exploitation of unpatched systems through the application of security patches and updates for operating systems, applications, drivers and firmware on Critical Cyber Systems in a timely manner using a risk-based methodology.” Cyber Vision automatically identifies device vulnerabilities and calculates risk scores so you can proactively build an improvement process to address risks.

Cisco’s capabilities, highlighted above, not only meet the current TSA Cybersecurity Directive requirements but also enable clients to deliver more robust cybersecurity capabilities to thwart efforts by industry threats. Most significantly, these capabilities are foundational for enabling both security and operational resiliency as well as optimizing the performance of mission-critical networks.

To learn more about how Cisco can help you secure your industrial operations, please contact us or visit cisco.com/go/iotsecurity. And don’t forget to subscribe to our OT security newsletter.

  Protecting the nation’s critical infrastructure requires strong cybersecurity controls. Being an industry leader in networking and security across both IT and OT, Cisco is uniquely positioned to help you comply with the latest TSA mandates. Learn what the requirements are.  Read More Cisco Blogs 

Cisco is aware of reports that Akira ransomware threat actors have been targeting Cisco VPNs that are not configured for multi-factor authentication to infiltrate organizations, and we have observed… Read more on Cisco Blogs

​

Cisco is aware of reports that Akira ransomware threat actors have been targeting Cisco VPNs that are not configured for multi-factor authentication to infiltrate organizations, and we have observed instances where threat actors appear to be targeting organizations that do not configure multi-factor authentication for their VPN users.

This highlights the importance of enabling multi-factor authentication (MFA) in VPN implementations. By implementing MFA, organizations can significantly reduce the risk of unauthorized access, including a potential ransomware infection. If a threat actor successfully gains unauthorized access to a user’s VPN credentials, such as through brute force attacks, MFA provides an additional layer of protection to prevent the threat actors from gaining access to the VPN.

Cisco has been actively collaborating with Rapid7 in the investigation of similar attack tactics. Cisco would like to thank Rapid7 for their valuable collaboration.

Akira Ransomware

Initial reports of the Akira ransomware date back to March 2023. The threat actors responsible for the Akira ransomware use different extortion strategies and operate a website on the TOR network (with a .onion domain) where they list victims and any pilfered information if the ransom demands are not met. Victims are directed to contact the attackers through this TOR-based site, using a unique identifier found in the ransom message they receive, to initiate negotiations.

Targeting VPN Implementations without MFA

When targeting VPNs in general, the first stage of the attack is carried out by taking advantage of exposed services or applications. The attackers often focus on the absence of or known vulnerabilities  in multi-factor authentication (MFA) and known vulnerabilities in VPN software. Once the attackers have obtained a foothold into a target network, they try to extract credentials through LSASS (Local Security Authority Subsystem Service) dumps to facilitate further movement within the network and elevate privileges if needed. The group has also been linked to using other tools commonly referred to as Living-Off-The-Land Binaries (LOLBins) or Commercial Off-The-Shelf (COTS) tools, such as PCHunter64, or engaging in the creation of minidumps to gather further intelligence about or pivot inside the target network.

Brute-Forcing vs. Purchasing Credentials

There are two primary ways regarding how the attackers might have gained access:

Brute-Forcing: We have seen evidence of brute force and password spraying attempts. This involves using automated tools to try many different combinations of usernames and passwords until the correct credentials are found. Password spraying is a type of brute-force attack in which an attacker attempts to gain unauthorized access to a large number of accounts by trying a few common passwords against many usernames. Unlike traditional brute-force attacks, where every possible password is tried for one user, password spraying focuses on trying a few passwords across many accounts, often avoiding account lockouts and detection. If the VPN configurations had more robust logging, it might be possible to see evidence of a brute-force attack, such as multiple failed login attempts. The following logs from a Cisco ASA can allow you to detect potential brute force attacks:
Login attempts with invalid username/password (%ASA-6-113015)
Example:
%ASA-6-113015: AAA user authentication Rejected: reason = reason : local database: user = user: user IP = xxx.xxx.xxx.xxx
Remote access VPN session creation attempts for unexpected connection profiles/tunnel groups (%ASA-4-113019, %ASA-4-722041, or %ASA-7-734003)
Purchasing Credentials through Dark Web Market: Attackers can sometimes acquire valid credentials by purchasing them on the dark web, an encrypted part of the internet often associated with illegal activities. These credentials might be available due to previous data breaches or through other means. Acquiring credentials in this way would likely leave no trace in the VPN’s logs, as the attacker would simply log in using valid credentials.

Logging within Cisco’s ASA

Logging is a crucial part of cybersecurity that involves recording events happening within a system. In the reported attack scenarios, the logging was not configured in the affected Cisco’s ASAs. This has made it challenging to determine precisely how the Akira ransomware attackers were able to access the VPNs. The absence of detailed logs leaves gaps in understanding, hindering a clear analysis of the attack method.

To set up logging on a Cisco ASA you can easily access the command-line interface (CLI) and use the logging enable, logging host, and logging trap commands to specify the logging server, severity levels, and other parameters. Sending logging data to a remote syslog server is recommended. This enables improved correlation and auditing of network and security incidents across various network devices.

Refer to the Guide to Secure the Cisco ASA Firewall to get detailed information about best practices to configure logging and secure a Cisco ASA.

Additional Forensics Guidance for Incident Responders

Refer to the Cisco ASA Forensics Guide for First Responders to obtain instructions on how to collect evidence from Cisco ASA devices. The document lists different commands that can be executed to assemble evidence for a probe, along with the corresponding output that needs to be captured when these commands are run. In addition, the document explains how to conduct integrity checks on the system images of Cisco ASA devices and details a method for gathering a core file or memory dump from such a device.

Cisco will remain vigilant in monitoring and investigating these activities and will update customers with any new findings or information.

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

InstagramFacebookTwitterLinkedIn

  Cisco has observed instances where threat actors appear to be targeting organizations that do not configure multi-factor authentication
for their VPN users. This highlights the importance of enabling multi-factor authentication (MFA) in VPN implementations.  Read More Cisco Blogs 

A deployment guide for wireless ThousandEyes agents deployed to monitor the Black Hat 2023 conference by Adam Kilgore & Ryan MacLennan

ThousandEyes (TE) Black Hat 2023 Deployment Guide

This guide… Read more on Cisco Blogs

​

A deployment guide for wireless ThousandEyes agents deployed to monitor the Black Hat 2023 conference by Adam Kilgore & Ryan MacLennan

ThousandEyes (TE) Black Hat 2023 Deployment Guide

This guide documents the setup and installation procedures used to deploy ThousandEyes at Black Hat 2023. This document covers configuration of the TE agent on Raspberry Pi 4, including both wired and wireless configurations. Needed hardware for the various setup functions are also listed.

Limitations of Deploying ThousandEyes Enterprise Agents on Raspberry Pi

Please note that TE Enterprise Agents will only show data for layer 3 hops, and will not show wireless access point data. This applies to deploying the TE Enterprise Agent on the Raspberry Pi and configuring it for wireless. The only TE agent that can provide wireless AP data is the TE Endpoint Agent, which is intended to be installed on laptops and personal devices.

Hardware Checklist for the Conference

There are several different hardware contingencies that must be accounted for before conference setup can take place.

A keyboard that can connect to the USB port on the Raspberry Pi
Laptops that can connect to the Pi Micro SD for TE agent provisioning. We used several different solutions for this:
An SD card to Micro SD adapter, which connected to the SD reader on a Mac laptop (the SD reader is not available on all models)
A Vivitar Micro SD to USB-C adapter, which connected to the USB-C port on a Mac laptop
A Micro SD to USB adapter that came with the Raspberry Pi 4 Cana Kit. This was connected to a USB to USB-C Mac adapter

Screens that can connect to the Raspberry Pi Micro HDMI port. This can be as simple as a normal HDMI monitor that connects to the Pi via HDMI to Micro HDMI, but these were not available during the setup days of the conference. We used two solutions:
A portable monitor that connected to the Pi via an HDMI to Micro HDMI cable
An adapter that connected to the Pi via Micro HDMI, connected to a Mac HDMI adapter, with the Mac HDMI adapter connected to a USB-C port on the Mac laptop

Pi Hardware Assembly

Black Hat 2023 Pi 4 Kits

10x Raspberry Pi 4 Cana Kits, each with the following:

32GB Micro SD card
Micro SD to USB adapter
Case
Heatsink
Fan

Example of built Pi with heatsinks and fan attached:

Note that we had overheating issues with the fan set to pull air out of the case. Orienting the fan to blow into the Pi case was more effective for our deployment.

Provision the ThousandEyes Image onto the Pi Micro SD

The Raspberry Pi can read Micro SD cards. The installation procedure is to download the TE Pi installer from the TE GUI, connect the Micro SDs to the provisioning laptops, and then use balenaEtcher to install the TE image to the Micro SD.

Retrieving the TE Agent Image & Documentation

From the ThousandEyes GUI, navigate to Cloud & Enterprise Agents > Agent Settings.

Click on Add New Enterprise Agent.

Locate the entry for Raspberry Pi 4 and click the Download – IMG button. The Installation Guide for the Pi install is also available from this page.

Connect the Micro SD(s) to Provisioning Laptops

If the provisioning laptops do not have a Micro SD card reader then adapter solutions will be needed to install the TE image to the Micro SD. For Black Hat 2023, we used a combination of the following hardware with Mac laptops.

2x Vivitar Mac USB-C to Micro SD adapter

1x Mac USB-C to USB adapter + 1x USB Micro SD Card Reader (included with some Pi bundles—the USB has an additional slot for the Micro SD card, which then plugs into a USB port—in this case, the Mac adapter shown).

1x Micro SD to SD adapter

Install the TE Image to Micro SDs

We used balenaEtcher to write the TE images to Micro SD, as covered in the TE Pi deployment guide.

Note that Etcher can write multiple images to multiple Micro SDs at once, your only limit is the number of Micro SD cards that can be connected to the provisioning laptops (USB-C, Micro SD card reader, USB port, etc.) at once.

Optional: Use Scripts to Provision Wireless that will Persist Across Reboots

Creating the wireless configuration for the Pi can occur either before or after booting the Pi for the first time with the Micro SD image that was configured in the prior section. If you would like to create the wireless configuration before booting the Pi, please follow the steps in this section. Alternatively, if wireless is not being used or wireless configuration can occur after booting the Pi, this section can safely be skipped. Please note that configuring wireless after booting the Pi will require enabling SSH on the TE agent, along with any requisite firewall rules to reach the Pi over port 22.

Also note that this section requires specific hardware and software, as covered in the next paragraphs. If the needed hardware and software is not available, the wireless configuration can still be accomplished using manual procedures given in later sections.

Before beginning the configuration, note that the SSID and SSID password must be hard coded. If the TE agent cannot connect to the hardcoded SSID after reboot, the wireless connection will fail. If the hardcoded SSID and password need to be changed after the initial setup, it will be necessary to (re)enable SSH on the TE agent.

Also note that the TE agent file system must be mounted onto a laptop to perform these steps. At Black Hat 2023, we were only able to accomplish this step using a Linux OS. Attempts to mount the Micro SD on Mac did not correctly mount the TE agent partitions.

Log in to a Linux system and connect a Micro SD with the TE agent installed (this procedure was covered in the prior section). Use the Micro SD connection methodology provided at the start of the guide.
Run the df command to verify the mounted partitions. The below screenshot shows the df output before and after connecting the Micro SD card, with the two mounted partitions visible after running the 2nd df command. Note the partition paths highlighted in red, as they will be needed to create the doit.sh script in Step 5.

Create a file named rc.local in a chosen directory. The specific name ‘rc.local’ is necessary for the script to run during the TE agent bootup process.
Create the script file:

# vim rc.local

Type ‘i’ to edit, and type or paste the following:

#!/bin/bash
/configure_te_pi.sh
exit 0

Type ‘:’ then ‘wq!’ to save the changes and exit the file.
Create the configure_te_pi.sh script in the same directory as the rc.local file. Replace the <SSID> and <SSID password> fields in the below script with the SSID and PSK that the Pi will connect to. The SSID and PSK can also be changed later, but this will require enabling SSH on the TE agent and then connecting to it.
Create the script file:

# vim configure_te_pi.sh

Type ‘i’ to edit, and type or paste the following:

#!/bin/bash
apt-get update -y
apt-get install te-va-unlock -y
apt-get install net-tools ifmetric wireless-tools -y
ifconfig wlan0 up
wpa_passphrase <SSID> <SSID password> > /etc/wpa_supplicant.conf
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf -D wext
ifmetric eth0 200
sed -i ‘1iauto wlan0niface wlan0 inet dhcp’ /etc/network/interfaces
dhclient wlan0
systemctl mask apt-news.service
systemctl mask esm-cache.service

Type ‘:’ then ‘wq!’ to save the changes and exit the file.
Create the doit.sh script in the same directory as the prior two scripts. Note that the paths to the /writeable and /system-boot mounts will need to be filled in with the paths collected in Step 1.
Create the script file:

# vim doit.sh

Type ‘i’ to edit, and type or paste the following, modifying the paths to match the output in Step 1:

#!/bin/bash
cp rc.local /media/<username>/writeable/etc
cp configure_te_pi.sh /media/<username>/writeable/etc
chmod +x /media/<username>/writeable/configure_te_pi.sh
chmod +x / media/<username>/writeable/etc/rc.local
umount /media/<username>/system-boot
umount /media/<username>/writable

Type ‘:’ then ‘wq!’ to save the changes and exit the file.

An example of the script created for the output in Step 1 is provided below.

#!/bin/bash
cp ~iredden/rc.local /media/iredden/writable/etc
cp ~iredden/configure_te_pi.sh /media/iredden/writable
chmod +x /media/iredden/writable/configure_te_pi.sh
chmod +x /media/iredden/writable/etc/rc.local
umount /media/iredden/system-boot
umount /media/iredden/writable

Run the doit.sh script. The doit.sh script will perform the configuration and then unmount the partitions, as shown below.

With the above steps complete, the Micro SD card can be removed from the laptop and connected to the Raspberry Pi. The Pi will execute the scripts during bootup and connect to wireless, if the SSID configured is reachable and the PSK is correct. Please see the next section for setup instructions and other considerations.

Also note that a wired connection to the Pi is still recommended for initial setup. Although the Pi TE agent can successfully pull a wireless IP address during initial bootup, the TE agent will not display a wireless IP address on a connected monitor, only a wired one.

Initial Setup of TE Agent

The Provision the ThousandEyes (TE) Image onto the Pi Micro SD section of this guide covered how to provision TE agents onto Micro SD cards. An optional section covering how to script wireless capability onto the TE agent image was also provided in the Use Scripts to Configure Wireless that will Persist Across Reboots section. This section will cover how to use the TE agent on the Micro SD card to perform initial setup.

Connect Micro SD to Raspberry Pi

Once the TE image has been installed on the Micro SD, connect the Micro SD to the Raspberry Pi.

Connect an Ethernet Cable to the Pi

It is necessary to know the IP address assigned to a TE agent after initial boot in order to complete initial registration. While a Pi with a wireless configured TE agent can pull a wireless IP during its initial bootup, it will not display the wireless IP on a connected monitor, only a wired IP address. For this reason, it is still recommended to connect an ethernet cable to a wireless Pi before initial power on and use the assigned eth0 IP to perform initial setup. The only alternative to a wired connection on initial boot is to identify the wireless IP address of the TE agent by means other than a connected monitor, such as via direct access to the wireless AP that the TE agent connects to.

Connect Keyboard and Monitor to Raspberry Pi

Two pieces of hardware are used to perform the initial setup of the Raspberry Pi TE agent: (1) a keyboard that can connect to the Pi via USB, and (2) a screen that can connect to the Pi via Micro HDMI. Initial setup can be performed without the keyboard if (a) an IP address is assigned to the TE agent via DHCP, and (b) the DHCP address can be identified via the connected screen or by other means. Similarly, initial setup can be completed without the screen if there is another way to identify the DHCP IP address assigned to the TE agent, such as admin access to the switch connected to the TE agent.

Despite the above alternatives, a connected keyboard and monitor are recommended for ease of deployment and any needed troubleshooting.

For the keyboard, we used a Bluetooth keyboard with a detached USB fob, but any keyboard that can connect to the Pi via USB will work.

For the screen, we used two separate hardware solutions.

The first solution is a Kenowa portable monitor with an HDMI to Micro HDMI cable. The HDMI side of the cable connects to the Kenowa, and the Micro HDMI side of the cable connects to the Raspberry Pi.

The second solution we used is an Elgato adapter that sits between the Raspberry Pi and a Mac laptop. The Pi connects to the Elgato via a micro HDMI to HDMI cable. The Elgato uses a regular HDMI cable to connect to the HDMI side of a Mac HDMI to USB-C adapter, with the USB-C side of the Mac adapter connecting directly to a Mac laptop.

Confirm Pi Hardware Connections

With the hardware configurations in the prior sections complete (i.e. the Pi is connected to an ethernet port, monitor, and keyboard), connect the Pi power adapter to begin initial boot.

Connect to the TE Agent GUI

Once the Pi is powered on, the TE agent install will take a few minutes to run (more if internet access is not available). Once the bootup is complete, the Pi will display a message if an IP address was assigned to the ethernet port via DHCP.

If no IP address was assigned to the ethernet port via DHCP, the TE agent will display a different message stating no IP address is configured. If this is the case, it will be necessary to either manually assign an IP address to the ethernet port using a keyboard, or identify the IP address of the wireless interface using admin access to the connected wireless AP.

Once the IP address of the TE agent is known, connect to the IP via HTTPS and sign in with username admin and password welcome.

Change the default password to something more secure and click Change Password.

The GUI should automatically take you to Agent in the left menu.

Keeping the ThousandEyes Physical Appliance GUI open, create a new tab in your browser and connect to the ThousandEyes web GUI, then click on Cloud Enterprise Agents > Agent Settings.

Click on the Add New Enterprise Agent

Copy the Access Group Token. Note: the Copy button may not work, in which case click the eye icon to show the token and copy it manually.

Return to the TE agent GUI and paste the Account Group Token into the Agent Click Continue.

The GUI will automatically load the Review Appliance Status and Diagnostics fields should change to green as the registration completes. Click the Complete button.

Click on the Network tab.

Configure the hostname for the agent.

Scroll down to the DNS configuration and enter the internal and External DNS servers. By Default, the agent will use the 208.67.222.222 and 208.67.222.220 external Umbrella DNS servers. These can be moved to DNS server 3 and 4 if internal DNS servers are preferred.

Scroll down and click on the Save Changes Note that the Applying network settings pop-up may get stuck. Reload the page to clear it.
Connect to the ThousandEyes web GUI and confirm that the agent shows as active under Enterprise Agents. For additional guidance, please see the section Validate Wireless Interface in the ThousandEyes Web GUI.

Manual Wireless Config Track

The Raspberry Pi TE agent can only connect via a wired connection using the default installation. If wireless is needed for the deployment but the necessary hardware/software was not available for the Optional: Use Scripts to Configure Wireless that will Persist Across Reboots section, or if the TE agent has already been deployed, the series of manual configuration steps starting in this section can be used to configure persistent wireless on a Pi TE agent.

Manual Wireless Config Track: Connect to the TE Agent via SSH

Documentation for connecting to a Pi TE agent via SSH can be found in this guide.

Manual Wireless Config Track: Jailbreak the TE Agent (to allow Wireless config)

Jailbreaking the TE agent is necessary for escalating to root and performing the commands necessary to build the scripts for the wireless configuration.

We should update the Ubuntu OS before running the jailbreak command. Connect to the TE agent via SSH, then run the following two commands to first update the OS and then jailbreak the install:

$ sudo apt-get update
$ sudo apt-get install te-va-unlock

With the TE agent successfully jailbroken, we can now escalate to root and perform the rest of the configuration.

Manual Wireless Config Track: Configure Wireless that will Persist Across Reboots using only the Pi

Before beginning this section, note that the SSID and SSID password must be hard coded. If the TE agent cannot connect to the hardcoded SSID after reboot, the wireless connection will fail. If the hardcoded SSID and password need to be changed after the initial setup, it will be necessary to (re)enable SSH on the TE agent and connect to it over SSH.

Escalate to root and create a file named rc.local in the /etc directory. The name is necessary for the script to run during the TE agent bootup process.

Create the script file:

$ sudo su
# cd /etc
# vim rc.local

Type ‘i’ to edit, and type or paste the following:

#!/bin/bash
/configure_te_pi.sh
exit 0

Type ‘:’ then ‘wq!’ to save the changes and exit the file. 
Run the following to set the script permissions:

# chmod +x /etc/rc.local

Create the configure_te_pi.sh script in the top level / directory. Replace the <SSID> and <SSID password> fields in the below script with the correct values. The SSID and password can also be changed later, but this will require a later SSH connection to the agent.
Create the script file:

# cd /
# vim configure_te_pi.sh

Type ‘i’ to edit, and type or paste the following, replacing the <SSID> and <SSID password> fields with the correct values:

#!/bin/bash
apt-get update -y
apt-get install te-va-unlock -y
apt-get install net-tools ifmetric wireless-tools -y
ifconfig wlan0 up
wpa_passphrase <SSID> <SSID password> > /etc/wpa_supplicant.conf
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf -D wext
ifmetric eth0 200
sed -i ‘1iauto wlan0niface wlan0 inet dhcp’ /etc/network/interfaces
dhclient wlan0
systemctl mask apt-news.service
systemctl mask esm-cache.service

Type ‘:’ then ‘wq!’ to save the changes and exit the file.
Run the following to set the script permissions:

# chmod +x /configure_te_pi.sh
Reboot the Pi and test to confirm that the wireless NIC can connect to the ThousandEyes web account.

Validate Wireless Interface in the ThousandEyes Web GUI

Confirm Wired and Wireless IPs

From the ThousandEyes web GUI, navigate to Cloud & Enterprise Agents > Agent Settings.

Identify a registered TE agent with a wireless interface, then click on it.

Locate the General Information The private IP address of the TE agent will be displayed, which is the IP address that the TE agent used to connect to the ThousandEyes web manager. If the TE agent has both an ethernet and wireless port, you can click the System Information link to confirm which IP address is assigned to each interface.

Confirm Wired and Wireless Interfaces for Tests

Navigate to Cloud & Enterprise Agents > Test Settings.

Expand one of the tests and then expand the Agents

Click on the Enterprise button and expand the dropdown next to an agent with wireless enabled. If the agent has both a wired and a wireless interface enabled, they will both be displayed as in the screenshot below.

Appendix

GitHub with scripts

ThousandEyes Wireless Setup GitHub

ThousandEyes Ports and Protocols

For TE agents to operate correctly, they will need to both connect to ThousandEyes over the internet and have firewall ACL access to perform any configured tests. Refer to the following document for lists of ports and protocols per region.

https://docs.thousandeyes.com/product-documentation/global-vantage-points/enterprise-agents/configuring/firewall-configuration-for-enterprise-agents

Script Functions Explained

Scripts to enable wireless functionality of the Raspberry Pi TE agent on startup are included in the Optional: Use Scripts to Provision Wireless that will Persist Across Reboots and Manual Wireless Config Track sections of this guide. This appendix section explains the script functionality line by line.

rc.local – this script will run as part of the TE agent startup process.

#!/bin/bash – initialize the script to run using bash
/configure_te_pi.sh – run the configure_te_pi.sh script
exit 0 – exit

configure_te_pi.sh – this script will configure the TE agent for wireless functionality

#!/bin/bash – initialize the script
apt-get update -y – update the underlying Ubuntu OS
apt-get install te-va-unlock -y – jailbreak the TE agent to allow additional sudo access
apt-get install net-tools ifmetric wireless-tools -y – install tools to allow wireless functionality
ifconfig wlan0 up – bring up a wireless LAN interface
wpa_passphrase <SSID> <SSID password> > /etc/wpa_supplicant.conf – configure SSID and PSK values for the wireless connection and write them to the wpa_supplicant.conf file
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf -D wext – imports the wpa_supplicant file and configures the wireless LAN interface to use the wext driver
ifmetric eth0 200 – set the wired interface to have a lower priority than the wireless interface
sed -i ‘1iauto wlan0niface wlan0 inet dhcp’ /etc/network/interfaces – configure DHCP for the wireless interface
dhclient wlan0 – enable DHCP for the wireless interface
systemctl mask apt-news.service – prevent apt-news.service from running. See the Wireless TE Agent Update Failure section for more information
systemctl mask esm-cache.service – prevent the esm-cache.service from running. See the Wireless TE Agent Update Failure section for more information

Wireless TE Agent Update Failure

During testing of the wireless TE agent for Raspberry Pi, it was noticed that the agents lost connectivity with the ThousandEyes web manager after a variable amount of time. Troubleshooting multiple failed agents identified logs associated with updates to the apt-news.service and esm-cache.service processes immediately before failure of the wlan:

syslog

Aug  6 02:17:04 registration-wireless systemd[1]: Started PackageKit Daemon.
Aug  6 02:17:20 registration-wireless systemd[1]: Starting Update APT News…
Aug  6 02:17:20 registration-wireless systemd[1]: Starting Update the local ESM caches…
Aug  6 02:17:21 registration-wireless systemd[1]: apt-news.service: Succeeded.
Aug  6 02:17:21 registration-wireless systemd[1]: Finished Update APT News.Aug  6 02:17:22 registration-wireless systemd[1]: esm-cache.service: Succeeded.
Aug  6 02:17:22 registration-wireless systemd[1]: Finished Update the local ESM caches.Aug  6 02:18:38 registration-wireless systemd-networkd[667]: wlan0: Lost carrier

More information on the two services can be found here:

https://askubuntu.com/questions/1452519/what-are-the-services-apt-news-and-esm-cache-and-how-do-i-disable-them

Masking the services so they are not run by systemd resolved the failure of wlan0:

$ sudo systemctl mask apt-news.service
$ sudo systemctl mask esm-cache.service

The above commands were then added to the startup scripts.

We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

InstagramFacebookTwitterLinkedIn

  Read how Validated Design TMEs deployed ThousandEyes on Pi4s wirelessly on the Black Hat USA 2023 network to provide network assurance and actionable insights.  Read More Cisco Blogs