So here is the deal.

We needed to set up a guest vlan in our network. We have
6 Aruba AP22 Access Points
1 Aruba 1930 Switch
1 Watchguard Firebox T45
1 Cisco router

Long story short I ended up Factory resetting all devices, mainly because we had have lost access to all devices except the firebox. Than I lost access to it to by disabling the trusted interface...

Anyways, Right now I can not get anything to work. Our office lost internet connection and my bosses are in my ass. I medelled with AI guides but it resulted in, well, nothing but problems.

I don't know if I am supposed to share my current configurations but I really need assitance mainly because I am not a Network Admin. I am a software developer and I have honestly no idea what I am doing or what I am supposed to be doing. (Don't ask why we do not have an IT department please)

If any of you could help me out or point me to the right direction, I would be gerateful.

EDIT:
So little clarification, we do not have a huge network, we practically had the devices and one VLAN that everyone in the company was able to connect to... No shared file storage or communication between devices just plain internet connection.

Then they ask us to create a guest network, we tried configurations but we realized that we needed an Aruba instant on account which the devices were somehow were already connected to. So we asked the Aruba support, they said we can not transfer the APs you'll need to factory reset all APs, so we did.

Then of course factory resetted APs were unable to connect to the internet so we thought we needed access to the switch, which was also set up by a third party as far as I know and they for some reason did not gave us the panel information.... So we had to reset the Switch to regain access.... So we did.

Finally firewall, it was all setup. But the damn AI guide made us do something without safety net and we lost access to it's interface alltogether so it resulted in this cluserfuck of situation.

2nd Edit: Why factory reset?

Aruba support team told us to do so. Config backup: we did not have access to neither Aruba switch nor Aruba APs. Why? This was a managed service at first.

Firebox reset, that was our ignorance.

This is a weird one for me....
I'm trying to help a restaurant with their networking issues which is affecting their ability to run the batch reports at the end of the night. It's also affecting the 3rd party POS providers ability to remote into their POS server, as well as mine to a different machine.

Their network is setup as such:
ISP modem > ISP router > Switch 1 + Switch 2 + NVR all in their own port on the router
Switch 1 runs all of their multimedia equipment (streaming devices, tvs, etc.)
Switch 2 just powers their POS router
Most of the POS stations are wired directly into the POS router, which also has a switch attached for the extra POS stations and the back office server.

Problem we are having: seems like intermittent network drops of some sort. My remote desktop tool says the device is available that I'm trying to remote into, but it keeps failing. The POS provider is having constant issues remoting into the backoffice/POS server. The batch report and night keeps failing.

From what the manager told me, the roku streaming devices for the TVs and the NVR (remote viewing) also keeps dropping connection.

They've called the ISP, ISP said there is no issue on their part (spectrum).

I however and starting to doubt that considering we are having issues with devices plugged directly into the ISP router (NVR). What can I do to try to CONFIRM where this issue is coming from so we can try to start getting it fixed?

UPDATE #1 5/16: I isolated the network last night, only running POS equipment from ISP router. Still having connection issues. I plugged a machine ONLY into ISP router that I was having issues remoting into, still having issues. I eliminated ISP router and plugged machine directly into modem, got straight in no problem. I then plugged POS router directly into modem, and everything worked amazing batch closed no problem. I left it that way and went in this AM to replace router with another, waiting to hear how batching goes tonight on new router. Saw

UPDATE 5/20/26: After replacing spectrums garbage router, all is good. Several nights flawless performance. Running the batch now goes through faster than ever, Rokus no longer repeatedly dropping signal. Etc… I had a feeling it was spectrums router in the beginning, just needed a way to prove to owner to get them to purchase their own equipment and switch.

I'm not interested in starting an argument and I do definitely have my options, but I'm genuinely curious to hear what people have to say.

I'm working for a new company, and in the year before I joined, they made a full system switch from Ubiquiti to Meraki. (Whether the move to Meraki was good or not, that's not what I'm interested in.) All of the team members talk about how bad Ubiquiti is. I come from an MSP where a fair number of our clients had full Ubiquiti networks with little to no problems. I'm just interested in what about Ubiquiti is problematic.

I WILL SAY, their old products had some problems... And the data breach they had in 2021 was... Not good (to put it lightly). I genuinely want to hear from others what your experience has been.

Edit: Yeah... I misspelled Ubiquiti. I'll handover my SSH keys, I'm to drunk to drive.

I am sort of new to Reddit but having access to so many other Senior Engineers makes me wonder what's the worst environments you've encountered?

I personally have run into massive multi-building, single vlan designs with >2000 hosts where STP was wreaking havoc on a daily basis but when I took it over was told "implementing VLAN's wouldn't fix this issue". Months later after implementing VLAN's on ancient HP Networking gear, that i was surprised support Dot1Q, was purring like a kitten. Then it was on to fix the next issue and the next and the next.

Funny how terribly built networks helps you understand at an extremely detailed level how STP/L2/L3 work. Funny how many engineers don't know the impact a TCN has on the normal operations. Sometimes the best way to learn the inner workings is to be exposed to these horrible network designs.

We have been facing issues with wifi in a warehouse for quite some time now I was able to get an older wispy DB spectrum analyzer and chanalyzer software.

We are using ubiquiti U6 LR AP's

Dedicated SSID for scanners only

Devices with the most issues are symbol TC70 scanners but the issues are not limited to the scanners and more modern devices such as Samsung S24 ultra still have similar issues.

TC70 behavior, constantly disconnecting from wifi

Very high latency pings.

Unifi shows the signal of the devices can be good at -67dBm while having a poor AP/client signal balance

RX rate 2-6Mbps

TX rate 65Mbps

(Not all of them are at this data rate)

Pings from hardwired desktop to scanner ranges from 50ms - 500ms

Requests sometimes timeout

Other times the pings spike to 2000-3000ms

Sometimes devices such as Samsung will connect to wifi but connect without Internet and then it will just start working.

Pings from switch > AP are sub 10ms

Pings from AP > TC70 scanners are anywhere between 28ms > 3000

We have tried turning 5ghz on the scanner SSID. We have tried changing MANY settings within unifi.

I just need some assistance at this point before this drives me mad issue has been going on for far too long at this point.

I have a chanalyzer file saved from walking the edges of the warehouse and being in the most problematic area.

Feel free to ask questions or provide assistance I can use any and all help at this point.

unifi dashboard

I am on site for a customer and my console cable is not in my bag. Amazon is saying next day delivered but I need it today unfortunately.

*Update*

I was able to get a 3 hour delivery (Amazon got it in 2) to a buddy's house. (weird because the customer prem location didnt have same day). Come to find out, this customer does not have their console user/pass and their network admin left their company. Fingers crossed that person is in a good mood and has that info. If not, looking like a long night waiting for a maintenance window.

A new windows update has broken NTLMv1. Any devices using EAP-PEAP will not be able to connect. A registry key must be changed to fix it.

• Open Registry Editor as an administrator.
• Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0.
• Create a new 32-bit DWORD value named BlockNtlmv1SSO if it does not already exist.
• Set its value to 0.
• Restart your PC and check whether the Wi-Fi connection works again.

https://www.neowin.net/news/microsoft-admits-windows-11-kb5127648-kb5126646-break-wpa-8021x-auth-shares-fix/

FYI

Migrating a buildings main internet connection from MPLS to VPLS. When changing the connection to VPLS and establishing the connection to my core switch I was able to confirm everything looked good. Routes looked good, could ping from switch to switch successfully... Success... But WiFi hasn't come back yet, that's odd, let me test the hard wire connection, weird, I'm not getting an IP address, so why is it I can ping across switches but suddenly DHCP isn't working?

Check my SVI's, check the VLANs and realize the VLANs don't align with the SVI's.. Then I realize these are the VLANs from my Core switch.. Check VTP status and it's configured... At this point there were many "fffuuuuuuuuuuuuckkk... fuck you VTP!!"'s

I disable VTP as I wish I had done before hand and quickly re-create all my VLANs to restore connectivity. Then I have to quickly move through the building to all of the other switches to recreate the VLANs.

So yeah, don't be like me, disable VTP because fuck you VTP.

EDIT: Also, if an another OS could be better than pfsense thats okay, as long as it does stateful firewalling

Hello everyone,

We are currently trying to reach 100Gbps with ours firewalls.

We have 2 ProLiant DL360 Gen10 with an intel xeon gold 6148 CPU @ 2.4GHZ wstuff with a Chelsio T62100-CR with a 100GBase-LR4 but it seems like we are running at 20Gbps at best.

I tried to tune my Chelsio by enabling hardware offload (checksum, large receive & TCP segmentation)

I feel like I'm missing something which is more system oriented.

Also I know it would be better to use a real hardware firewall but we are small volunteer organization with low budget.

Thank you for your help.

Is there any tool as described in title for enterprises? a very specific LLM could easily point out "strange" things in a large pcap

I'm currently troubleshooting a network issue where laptops on the company network are receiving IP addresses from a DHCP scope associated with the Telephony VLAN instead of the Admin VLAN.

No changes have been made in the data room, on the switches, or on the network configuration to my knowledge. However, new devices are consistently being assigned to the wrong VLAN.

Could someone explain what could cause this behavior? What should I check first to identify whether the issue is related to switch port configuration, DHCP relay settings, VLAN assignment policies, or another network component?

Any insights or troubleshooting suggestions would be appreciated.

I am an IT Generalist who wants to specialize and is about 40 labs into the CCNA using Jeremy IT course.

Today I just realized that the biggest reason I feel like im acing through the protocols and not having a hard time troubleshooting is because I am being given network topology diagrams where I can quickly see what's connected to what AND quickly access the CLI by just clicking on the device icon from the diagrams.

From my understanding is that this is not real life. You have to individually connect to each device one by one with a console cable and use commands like sh run/tracert to have an idea what the hell is going on. From my readings the most popular advice in this sub is the ability to draw a picture/diagram in your head or paper while troubleshooting, while this seems valid it also feels very time consuming and prone to errors.

Hello everyone. I am an MSP that works closely with a university. THIS IS NOT A HOMELAB.

We have two Esports labs with 24 endpoints per room. In most games, the latency spikes and becomes unplayable for our players. The network is as follows:

1. Unifi EFG - WAN connection is 5G, SMF, DHCP
2. Unifi Aggregation - 2x10G uplink, 2x10G downlink, LACP
3. Unifi Pro Aggregation - 1G to each workstation, 2x10 uplink, LACP
4. Workstation - Intel X520-DA2, Intel SFP+, OM3

Completely separate from campus network
1x5G symmetric WAN
Smart Queues and Flow Control off
RSTP on, root is Aggregation (4096)
MTU 1500
IDS and IPS turned on
Separate management and workstation VLANs

I’m seeing about 2.5% packet loss and players are seeing +200ms ping spikes.

I think it’s a buffer issue (but I am obviously not sure). Unifi switches seem to have minuscule (16MB) buffers. My next step is temporarily putting in an Arista 7280 using LANZ to check.

I understand that yeah, these guys are playing video games but this is a legitimate workload on hardware that costs more than some small/medium business networks.

I’ve got an issue here that has several folks stumped. I noticed an issue with slow internet download speeds over a month ago after some network changes upstream. My main vlans were only seeing about 40mbps to Speedtest.net and similar sites. It looked like some firewall bottleneck or QOS issue, and I reported it upstream, they could find no problem. After more troubleshooting I found a subnet here that was seeing proper 900mbps or better speeds, sitting behind the same pair of core switches. More troubleshooting, and I found more at normal speeds. Then came the big break, only vlans that existed on my core switches were slow. Any vlans downstream that were IP routed were fine.

This is a lab environment, basically a small data center. Pair of SONiC switches in mclag setup. Downstream area switches using IP routes back to the cores. TOR switches connected to the area switches. Just a small number of vlans exist on the Core switches, my main infrastructure vlans and a bench vlan.

As a test, I eliminated the infrastructure vlans from the core switches and replaced it with an IP route to the downstream area switches, and to my surprise the speed problem disappeared. Gateway IP was already on the downstream Area switch, that didn’t change.

I set up a new vlan on a single core, with a single access port again on the core. Laptop connected on that vlan saw slow transfer speeds to the internet.

Set up iPerf on some local systems, speeds between core vlans were 900mbps or better, between local IP routes were normal. So everything from the cores on down seem fine, it’s only going upstream.

The change a month back upstream changed to a different pair of switches, and changed OSPF to BGP routing. I have no control or visibility to the upstream side config.

I did a few packet traces but didn’t see anything unusual, no excessive retries, etc.

I’m really stumped on troubleshooting this. To me everything screams bug in SONiC’s vlan handling, but it seems I’m the only one seeing this issue. Dell Enterprise SONiC for what it’s worth, and I’ve engaged their support engineer who is also stumped.

EDIT 1: Forgot to add, running a ping test from outside my network to a system on a problem vlan, pings generally were about 2ms without speedtest running, but spiked to 130-190ms when speedtest ran.

I will preface this by saying I am not a network engineer however I do mess around with a lot of networking equipment in my home lab.

By trade I am a systems engineer.

In one of the data centers that I work on we had a OSPF routing issue that the network engineer determined was due to a flapping interface.

In this design there is a distribution switch that is dual homed to two disparate core routers. Each connection from the distribution switch is a single link. The core routers have a single link between them.

Yesterday started getting alerts that some of our servers were unavailable. However they were up it was just an issue of connectivity between specific subnets in the network.

Our network engineer diagnosed the issue as a flapping link to core router one. According to him this is the shortest path via OSPF. However the link was flapping a lot so OSPF was apparently in a bad state as routing tables constantly trying to determine the correct path and whether to route packets to core router one or core router two.

I asked why do we not have two connections to each core router from the distribution switch. Port channels is the terminology that I am familiar with from messing around with Cisco but these are juniper devices so I think it's called something different.

The network engineer told me that if we had LACP connections to each core router the issue would have actually been worse. As one of the links to router a would have been fine and one of the links to router would flap.

Honestly I don't know enough about networking to prove this person wrong. However I've worked in Enterprise data centers enough and I'm with enough Enterprise data center network engineers to know that they have redundant links everywhere.

I'm trying to figure out if this network engineer just doesn't know how to do what I am asking him to do.

I want redundancy everywhere, redundant links between all devices, and redundant paths such as a distribution switch having the ability to route to two different core routers.

If I go to Google Gemini or chat GPT and provide the network engineer feedback, I'm always told it's just an AI hallucination.

Beyond redundant links to each core router. I would think that there should be some sort of OSPF configuration that would blacklist the flapping connection. If we were using whatever juniper's equivalent is for ether channel, I would think that ideally we'd never even have to have a routing table change and we would just know that we have a singular bad link and we need to troubleshoot transceivers and/or cables.

I'm also not 100% sure that I believe that the constant flapping of the interface was the core issue. My understanding is that OSPF converges pretty fast and that even if it link was flapping that we wouldn't lose connectivity as long as both of the northbound core routers or configured properly.

Can someone provide me some insight here, and possibly make some suggestions to how I should approach better resiliency and redundancy to the network engineer.

Sorry for the long post I didn't know how to make it much shorter than I did.

I am setting up a laptop as a field tech laptop. What are some other opensource, free or low cost apps I should consider?

I will be adding wireshark, Angry IP scanner, Netspot (heatmap), Fing, putty, AnyDesk, Unifi software, and whatever else i can think of. What are some applications that have helped you for work and troubleshooting networks in the field?

I'm having an issue with getting an IPSEC VPN working over Starlink and could use a little bit of help pointing me in the right direction.

The head end device is a Cisco Firepower FTD1140 and the client is a Cradlepoint R1900 mobile router. I have set up a route-based VTI VPN between the FTD and the Cradlepoint and the VPN is established.

Here's where I am having the issue - everything works exactly as it should when using the internal cellular modem. VPN comes up, routes build correctly, and everything is fine. I have the Starlink set up so the users can flip a switch to power it on as a backup internet source. The Cradlepoint sees the WAN link come up and switches to use that instead of the cell modem.

From everything I can see, the VPN link comes up and establishes correctly. Tunnels report as up on both the Cradlepoint and the FTD side, but no traffic passes at all. From a client computer, I can ping the IP address of the outside interface on the FTD, but that is the only thing that responds.

I have tried setting the VPN adapter MTU on the Cradlepoint to 1300 and it still does not pass data. Turning the Starlink off restores all communications.

Would this be an issue with CGNAT? For some reason, the FTD does not think that the connection needs NAT-T enabled. The IP address reported by the Cradlepoint does not match the address that the FTD reports. The Cradlepoint is set to use "Force UDP Encapsulation" but the FTD reports "IKEv2-PROTO-7: (605): Process NAT discovery notify IKEv2-PROTO-4: (605): NAT-T is disabled".

Any tips or tricks on getting this VPN running over Starlink?

We have two routers set up for redundancy, one on MPLS and one on a backup internet circuit. The MPLS BGP session has been dropping randomly for the past few weeks, but only between about 9am and 6pm on weekdays, never at night or on weekends. It comes back on its own after 20-30 seconds, but it's enough to trigger a failover to the backup line and cause complaints from the office.

I've checked the obvious stuff already. No errors on the interface, CPU on the router stays low even when this happens, hold timers match on both sides, and the carrier says their side looks fine too. The debug logs just show the session going down from a timeout, nothing more specific than that.

Has anyone run into something like this before? Trying to figure out if this is a traffic or congestion issue tied to business hours, or something I'm missing on my end. Also wondering if it's worth pushing the carrier harder before I start running packet captures all day.

2 New switch stack of cisco 9300 switches with fiber uplink on each in active/active config. The uplinks were flapping constantly when 1 specific pc was connected. The pc was frozen and is no longer connected. On the older switch there was no problem on the 9300 as soon as the device was connected the uplinks started to drop. For my own sanity how can single windows device on an access port with broadcast and bpdu guard on cause issues with the uplink port?

HELP! been dealing with a weird one for a week and could use some outside perspective.

Setup:campus network, around 2000 users. Single firewall fortigate running dual-WAN out to two separate fiber lines, through SD-WAN. Core is a pair of switches Fortiswitch, feeding more than 60 access switches BDCOM downstream, plus a wireless controller ARUBA handling about 200 APs across the buildings.

The problem: multi-hour internet outages with no pattern I can pin down , not tied to time of day, not tied to load. Rebooting the ONTs clears it every time, but only for something like 45 minutes before it starts degrading again and eventually drops fully.

Checked spanning tree and hunted for a physical loop , both came back clean. Kind of stuck on where to look next. Anyone run into something that behaves like this?

UPDATE: thank you guys for all the support, i’ve been reading the comments but unable to respond right now.
I’m kinda frustrated since I’ve been working on this issue alone for a week, the site is too big and got lost with what should be done.
Back from the start and trying to rule out the existence of a loop, shutting down every switch in each department.
This morning i found as usual that one ONT is completely down, can’t ping from my laptop, nor from fortigate (can’t pint to the ONT nor internet)
Fortigate is still able to ping the second ONT, but all vlans passing through this ONT have no internet as well, and I also have an internal application not working or really slow to load.

I have eliminated half of departments until now but still no luck.

UPDATE: Thank you for all the input. I actually found the culprit at last, it was the firewall. turned out that having an application filter with the free version of FortiGate does mess up with the network, i did notice that when i was first recruited and wanted to secure the newtork and deploy every security aspect, when i was configuring the web filter or the DNS filter, the network went down instantly.
But with the application filter, it just deteriorates over time.
Since my recruitment, I've requested many time the administration to but the license but I guess it's not one of their priorities.

UPDATE: This has been pretty much solved! There were two primary issues. One is that on boot the intel drivers were not loading correctly and the device was entering safe mode. This was reducing our total number of interrupts to one. In addition to that, we had to create a large RAM buffer and bypass the page cache using direct I/O. We also split the packets up across several ports so each thread is reading from its own port, but I’m not entirely sure this was necessary. Thank you all for your suggestions!

Background: We have a connection which is streaming ~9000 byte jumbo packets directly from a 100 GbE switch to a server (Red Hat Linux). The data stream is around 40-45 gigabit of continuous data, and we are attempting to receive the packets and immediately store the data into files with no processing. Currently, we have multiple threads (6 or so) that essentially round robin the packets and store to their own files, then merge the files after the data transfer is complete.

Problem: It seems that our NIC buffer is filling up, and we are only getting around 20 GbE (or less) after this occurs. We have tried pretty much all of the suggestions from the Red Hat guides, and on paper, our specs seem that they should be able to handle this data, but is there something special we need to be doing to achieve higher speeds?

I am not able to provide specific details regarding the switch or server for security purposes, but I can provide the following (somewhat vague) details:

Processor: >80 cores @ 2.25 GHz

RAM: 16x32 GB PC5 DDR5 ECC RDIMM

Storage: Micron 7500 PRO PCIe 4.0

100 GbE Adapter: Intel 100-GbE Network Adapter PCIe 4.0x16

Additional (maybe relevant) Components:

Broadcom HBA 9500-8i PCIe 4.0 x8
10 GbE Ethenet Adapter PCIe 3.0 x8

Do any of these components act as bottlenecks in storing the data, or is there a faster way to retrieve the data from the NIC than just opening a socket a pulling the data with multiple threads?

Some of our troubleshooting has involved increasing the ring buffer size, increasing the default and maximum rmem and wmem values (and a few other things in the Red Hat guide).

I inherited a VXLAN EVPN network and it has been pretty stable until recently. From time to time, the layer3 or at least the inter-vlan drops for a few seconds. It happened last week, yesterday and today.

For what I can tell, the layer2 is fine. The VLANs gateway is on the service leafs (vPC pair). We are using ePBR to force the inter-vlan to the firewall, which is connected to the service leafs, the firewall will route the traffic back to the service leafs.

This has been stable for several months and no network changes. At this point, I'm not sure if the Nexus pair (where the SVI with ePBR) is causing the network downtime or the firewall.

We recently migrated to OpenShift for virtualization and containers. This was a month or two ago.

The firewall has a static route to the 172.16.0.0/16 with the next-hop of the HSRP VIP of the vPC pair. The service leafs are learning the 172.16/16 from a Catalyst leaf. All the VLANs are stretched to the service leafs via L2VNI.

```

[Fw]---/29---[service leafs pair]---[spines]---[cat9k leaf]

```

I know the layer2 (or L2VNI) didn't fail because Zabbix is not reporting any ICMPloss between it and the hosts in the same VLAN. However, any subnets within the same VRF drops for a few seconds.

The leafs CPU utilization history is basically idle 1% - 3%. The firewall is the same thing. Network utilization wise, we are using less than 800Mbps of traffic and the network is 40Gbps links.

I'm trying to get some ideas what I should look for to identity the issue and what is causing it.

My nexus version is 10.5.4 and the Catalyst is 17.15.4b. The Palo Alto firewalls (active/passive) are version 11.14.0.

Thanks.

After a switch firmware upgrade, users started having intermittent wireless connectivity issues, with brief disconnects and reconnects while connected to the corporate Wi-Fi network. Multiple access points are repeatedly losing and re-establishing their CAPWAP connection to the wireless controller. While the affected APs remain powered on and do not reboot, their CAPWAP association uptime continuously resets, causing APs to temporarily enter FlexConnect standalone mode before rejoining the controller. Controller logs show repeated DTLS tunnel disconnects, while AP logs indicate CAPWAP restarts due to missed data keepalives ("data keepalive not received").

I ruled out AP reboots, PoE issues, interface errors, STP. Physical switch ports are operating normally with no CRC, discards, or link errors. Opened Cisco TAC case but they're taking long so looking to the masses for help. I'm hoping someone experienced something similar that can throw in their two cents.

How's the impact treating you?

I've been in a call since 1:30 am and still going as I write this post.

I have two sites, A and B - connected by a S2S IPSec VPN on gigabit links.

Site A has a Fortigate 400E running latest v7.2.
Site B has a Fortigate 120G running latest v7.6.

Site B is able to line-rate on iPerf3 to A on TCP/UDP.
Site A is able to line-rate on iPerf3 to B on UDP only.

TCP is very slow (less than 1% of UDP).

I have the same config on both sides. VPN interface(s) have tcp-mss set to 1418 on both sides. No profiles applied to impact performance. DH is 21 w/ AES256GCM-PRFSHA384 if it makes any difference.

What am I missing here?
Thanks, real head scratcher.