What happens when someone types google.com into a web browser? Pt. 2

Continuation of Part 1 – https://mickx009.org/2020/03/22/what-happens-when-someone-types-google-com-into-a-web-browser-pt-1-dns-arp/

Per part 1, the computer in question now has received an answer to what public IP address Google.com maps to. With this information the computer can now begin a request to the server. The computer web browser will start by sending a request down from Layer 7 (application) to its layer 4 TCP stack requesting a socket stream with the destination port of TCP 443 (HTTPS). The TCP stack begins creating a segment with the destination port of HTTPS and source port from the OS random pool.

The TCP segment is then sent down to Network/Layer 3 where an additional IP header is added to the TCP segment. The IP header will contain both a source and destination IP address. The destination IP is the public IP the DNS query received mapping to Google.com. The source IP will in theory be the IP address assigned to the computer trying to reach Google.com. After the IP header is added to the segment, the networking stack then sends the now IP packet down to Layer 2 where a Frame header is added on. The Frame header includes the source MAC address, which is the hardware address associated with the computer NIC, and the destination MAC address that is assigned to the default gateway. Part 1 showed that the computer now knows the MAC address of the default gateway due to the ARP broadcast process. If the local computer for some reason does not know the MAC address of the router/gateway, then the ARP process would need to be run again. From here the now Frame is sent to the local router/gateway where the Frame header is taken off the IP Packet. The router then processes how to reach the destination IP of the packet via the device’s routing table. Most likely a Default Route entry is found in the routing table which tells the router where to send the incoming traffic. Before the computer Packet is sent off, the router first will add another Frame on top of the Packet. The Frame will once again have a source and destination MAC address. The source being the WAN side of the router MAC address, and the destination being the MAC address of the next hop router interface. The hopping between Layer 3 and Layer 2 will continue until the destination is reached. And ultimately continue in both directions until the computer sends and receives all necessary payload.

It’s worth mentioning that in a typical Enterprise/Residential network, internally the hosts are using IPv4 RFC 1918 address space. This address space is not usable on the public internet, so most internet users require their router to perform NAT to reach the IPv4 internet. At the edge of the network before reaching the internet, a router will typically perform a 1 to many Network Address Translation. This means all internal RFC 1918 hosts will get translated to one public IP address assigned to the WAN/public side of the router. All traffic internet bound will appear to be coming from that one public IP. Sessions are managed via internal and external source/destination layer 4 ports.

When the computer in question reaches the server hosting Google.com, the server receives the Frame, removes layer 2 and layer 3 headers, then processes the segment on the server’s listening TCP port, in this case 443. The received segment has the SYN flag set which starts the beginning of a TCP connection. The server responds back to the computer with the SYN ACK flags set, then receives a response back from the computer with a final ACK. The final ACK is the end of the initial three way handshake and data can start getting transmitted between the server and computer.

TCP Selective Acknowledgement – NOTES

TCP Selective Acknowledgement (SACK) is a TCP option that works as an alternative segment recovery/retransmit to the normal cumulative acknowledgements and 3 duplicate ACKs rule. If both client/server support the option then one will see SACK Permitted in a packet capture during the initial three way handshake.

When a segment is lost during a TCP connection the sender will continue transmitting data in line with the allowed sender window. When SACK is enabled the receiver will append a normal ACK of what’s received with the SACK option, telling the receiver what blocks of data have been received beyond the missing segment. The receiver sends the SACK with what’s called the Left Edge of the data block and the Right Edge of the data block. This tells the TCP sender that the missing segment is the block of data (segment number) before the left edge of the SACK Option block. The SACK option is more efficient over the wire because it does not require the sender to retransmit segments that have already reached the receiver. The 3 duplicate ACKs requires the sender to retransmit from the missing segment to the point at which the 3 duplicate ACKs were received.

TCP Fast Retransmit – Fast Recovery – NOTES

Fast Retransmit/Fast Recovery:

When a TCP connection has been established and TCP segments are being sent/received, the two involved nodes keep track of how much data has been sent through Sequence and Acknowledgment numbers.  If the receiver finds an out of order segment has arrived (wrong sequence number from sender) then it will immediately send a duplicate ACK back to the sender.  This duplicate ACK occurs to let the sender know about the out of order segment.

The sender will not know immediately whether the duplicate ACK is due to reordering of the segments/stream or if there’s traffic loss happening over the connection.  The sender in this scenario will wait until it receives 3 duplicate ACKs back from the receiver to deem there is actual segment loss.  Once this happens the sender will set the Slow Start Treshold (ssthresh) to half the current send window, Congestion Window (CWND), and then retransmit the missing segment.  When the segment has been retransmitted the sender then sets CWND to the current ssthresh + 3 maximum segment sizes (mss).  The addition of 3 mss is to inflate the window to suffice the segments that have already reached the receiver’s buffer.  From here on every duplicate ACK received by the sender will add an additional 1 mss to the window.  Also the sender will attempt to transmit a new packet if the CWND will allow it.  

When the next ACK arrives to the sender that is for actual new data and not duplicate, the sender sets the CWND to the current ssthresh.  Per previous notes, if CWND and ssthresh are the same then the TCP connection is in the congestion avoidance phase again.  In the Congestion Avoidance phase the CWND is increased by 1 segment only if ALL segments in the window have been acknowledged by the receiver.  The Congestion Avoidance phase runs to find out the capacity of the connection.

  1. 3 Duplicate ACKs are received and Fast Retransmit is Kicked off.
  2. Set ssthresh to half the current send window and retransmit the missing segment.  
  3. Sender sets Congestion Window to value of ssthresh plus 3 mss for segments already in receiver’s buffer.
  4. Transmit a new packet if allowed by Congestion Window and add 1 mss per each new duplicated ACK.
  5. Once ACK for new actual data arrives to sender, set Congestion Window to same value as ssthresh, continue Congestion Avoidance phase.      

What happens when someone types google.com into a web browser? Pt 1. DNS & ARP

I realized that I’ve been asked the question What happens when you type <insert website> into a browser? in a lot of interviews, and I’m not sure I’ve ever answered well. I always fumble my way through it and realize I’m thoroughly missing steps. I’m going to try and answer this from the perspective of interviewing for a Network Engineer/Admin position, so more networking focused than anything else. I’m also going to operate under the assumption this is IPv4 and there are no local caches, MAC table entries, etc. Surely I’ll miss some steps, these can probably be some ongoing write ups that get updated over time.

When someone enters google.com into a web browser the first thing from a networking perspective (besides OSI Layer 1) that needs to happen is IP address resolution. The computer needs to know what IP address the website has and this is found through a DNS query. The computer’s network stack will have a DNS client configuration that points to a known DNS server. Lets say for this example the computer has the server 8.8.8.8 configured for DNS. In this case, the computer will attempt to send a DNS query to the DNS server at IP address 8.8.8.8 over UDP port 53. The layer 4 protocol and port number are not a guarantee but currently more often than not UDP 53 is what will be used.

On both residential and enterprise networks we typically see RFC 1918 address space – ie. internal, non public space. In this example lets assume the PC has the internal IP address of 10.30.1.50/24. When the computer needs to reach an IP address that is not on its local subnet (10.30.1.0/24), it will send its traffic to whatever IP is assigned to its NIC’s Default Gateway (10.30.1.1). Because the IP address 8.8.8.8 does not fall into the IP range of the local subnet, the computer will try and send its DNS query to its default gateway at 10.30.1.1.

Moving from IP (layer 3) to Ethernet (layer 2), for a computer to forward datagrams and frames over an ethernet network it needs to know destination MAC addresses. In this scenario the PC needs to forward its DNS query to the router/layer 3 device that has the assigned IP 10.30.1.1. To find this MAC address the computer will send out what’s called an Address Resolution Protocol (ARP) request. The ARP request is a layer 2 broadcast that gets forwarded to every device in the layer 2 broadcast domain, such as a VLAN. The layer 2 ARP broadcast (ff:ff:ff:ff:ff:ff) is essentially asking every device ‘who has the IP address 10.30.1.1?’, and the only device that responds to the request is our router that allows access out to the public internet.

Once the router/default gateway responds to the ARP request the computer is then able to send out a DNS query to the public internet which routes the query to the nearest point of presence for 8.8.8.8. Public DNS servers like 8.8.8.8 or ISP provided DNS servers are typically called Recursive DNS Servers. If the Recursive DNS server knows what the public IP address is for ‘Google.com’ then it immediately responds back to the computer with the answer. If the recursive DNS server does not know then it will send a request to one of the Internet’s 13 Root Nameservers. The root nameserver will respond to the request with one of the Top Level Domain (TLD) Nameservers, depending on the end of the domain name – ie. .com, .net, etc. The TLD nameservers will then respond to the recursive server with the final Authoritative Nameserver that knows what IP address maps to the domain Google.com. The Authoritative Nameservers are DNS hosting providers such as GoDaddy. Once the recursive DNS server receives the IP address it then caches the information so that the DNS query process from the client is answered right away the next time.

Now that the computer has successfully received an answer for the DNS query, it now can start the process of requesting data from the server hosting Google.com.

FortiClient SSL VPN – DHCP

Someone reached out recently and told me that a Fortinet Fortigate SSL VPN was acting up and DHCP was not working correctly. This person was receiving the Windows error message on their PC while working remote that there was a duplicate address problem.

The problem I believe wound up being something on that person’s home internal network, but I did attempt to look into the issue right away and could not find a lot of information on DHCP leases for the Fortigate SSL VPN IP range. As any Fortigate admin knows, one can log into the GUI and go to Monitor–>DHCP Monitor, or Monitor–>SSL-VPN Monitor. From there you can view all DHCP leases (if you’re using the firewall as a DHCP server) or view all active SSL VPN connections.

GUI SSL-VPN Monitor can be viewed in CLI via below:
#get vpn ssl monitor

I never thought about it before but I assumed I could see DHCP leases for the SSL VPN IP range in the DHCP monitor window, but there was nothing when I tried. Under the SSL VPN monitor however I could see numerous connections with valid IPs for the VPN range.

I looked into this a bit to find DHCP lease information for the VPN and apparently the DHCP daemon does not actually hand out IPs to VPN clients. The VPN clients get IP address information from the sslvpn daemon itself. DHCP options such as lease time do not exist because of this. The SSL VPN DHCP lease time is essentially the time of the VPN connection. Once the VPN connection is removed, that IP goes straight back into the IP pool for the next incoming SSL connection.

Seems somewhat obvious after typing this out, but still glad I did.

FortiClient SSL VPN Timeout

Due to the global COVID-19 pandemic there have been many people switching from working in the office to telecommuting at home. This has required many individuals who are not typically used to working remote get used to operating a VPN client.

An issue we kept running into was a report that every day at a specific time the FortiClient VPN connection would drop. The odd thing was that this specific time of dropping was the same time for the same person, but not the same time across all users having the issue. Eventually we traced the issue back to someone who started work everyday at about 5:45 AM and everyday the connection would drop at 1:45 PM. This lead us to find out what the default SSL VPN timeout setting was for Fortigate SSL VPN access. Per below, the default timeout setting for an SSL VPN client was 28800 seconds – ie. 8 hours.

After some discussion we decided to increase the timeout value to 43200 – 12 hours.

Once the commands were entered on the Fortigate above these disconnect reports went silent.

Fortigate Link Monitor – (Cisco IP SLA Equivalent)

In an office or branch location that relies on internet access for productivity, it’s obviously typical to see a primary and secondary internet connection from two separate providers. In IPv4 world if you are not using your own IP space and BGP peering with the upstream providers, then automatic failover when the primary connection goes down becomes a concern. In a simple active/passive two ISP to one router (dual homed) setup, the router/firewall will have two static default routes for each provider. Each route will have a weight or metric determining which one is more preferred. Active/Active connections is an option, especially now with the rise of SD-WAN solutions, but often times a simple active/passive is what’s needed. In Fortinet world the metric for active/passive is Distance. The lower the Distance the more preferred the route.

In the image above the primary ISP route will be used due to the distance value of 1. But how does the firewall know when the Primary ISP is having issues and needs to stop sending internet bound traffic that way? With this configuration by itself, the only way the firewall will remove the default route to our primary ISP is if the interface on the device itself goes down. As most network operators know, it’s very common for a physical interface on a router/firewall to stay up, but the modem or somewhere upstream is actually not working. What we really need to do for this situation is setup what Fortinet calls a Link Monitor (previously called Dead Gateway Detection).

The function of the Link Monitor is to take an interface and continuously try and call out to an IP address up stream. ‘Call out’ to an IP address means ping, tcp/udp echo, or http query. But the general idea for this scenario is if the Fortigate can access something upstream then the internet connection must be alive and well. If after the specified link monitor failure attempts occurs, then the firewall will either shut down the Primary ISP interface or simply update the routing table. In the configuration example below the firewall is set to ping 8.8.8.8 out the Primary ISP interface.

SETTINGS OF IMPORTANCE:

set interval 2 = Time in seconds between sending link health check packets. Set to 2 seconds.

set failtime 5 = Number of times a health check can fail.

set recoverytime 10 = Number of times health check must succeed to verify connection is back up.

set update-static-route enable = Removes static route from routing table if link monitor fails.

In our dual homed example the Fortigate sends a ping to 8.8.8.8 out WAN1 connected to the Primary ISP every 2 seconds. If the ping fails to reach 8.8.8.8 five times in a row then the default static route is removed from the firewall routing table and the secondary default static route takes over. Once the ping succeeds over the Primary ISP interface 10 times, the default static route is added back.

The configuration above is what I’ve used in the past successfully which differs from the Fortinet Link Monitor defaults. With the default settings I’ve had issues with flapping between connections. Every situation is different but the configuration above for internet connections is what I’ve had the best luck with. The Link Monitor feature is what Cisco world calls an IP SLA.

Floating Static Route – Backup Default

The floating static route is a quick tool for directing traffic to destinations a different way through the network when default metrics work against us or there is a link failure.  In conjunction with dynamic routing protocols, a network operator can use the floating static route and route table Administrative Distance to manipulate traffic flows.  

Branch Office

In the diagram above you’ll see a small OSPF area with two exit points to the internet.  The business has asked you as the network operator to have the primary route to the internet for the branch via R2 –> R3 –> R4.  The secondary or backup internet connection is accessed via R2 –> R1.  Lets say all links are equal speed/cost in terms of OSPF metrics and both internet gateways are advertising their respective default quad 0 (0.0.0.0/0) route into the OSPF area.  If that’s the case then due to OSPF cost value R2 will install the secondary default route into its forwarding table instead of the primary.  An option to direct internet bound traffic to R3/R4 would be to have R1 stop advertising its quad 0 into the OSPF domain, but then R2 doesn’t know about the secondary connection when the link between R2 and R3 fails.  A quick way of getting around this issue is by utilizing a floating static route on R2.

By default a router uses Administrative Distance (AD) to choose best path through a network when there are two routes to the same destination. The AD value of a route is typically dependent on how the routing device received the route entry, and the lower the value the more preferred the route. The typical default routing protocol AD values are below, although this is not exactly the same for all vendors.

Administrative Distance Values:

  • Static – 1
  • EBGP – 20
  • EIGRP – 90
  • OSPF – 110
  • iBGP – 200

Going back to the diagram above, if we need R2 to have a backup quad 0 route to R1, we can add a static route pointing to the next hop assigned to R1. But per the AD values list, a static route entry has a value of 1, which is preferred over the dynamic protocols. To get the static route to function as a less preferred option the route entry needs to be added into the route table with an AD value higher than the quad 0 routing table entry advertised by R3 through OSPF. OSPF has an AD value of 110 by default, so modifying the static route to the secondary internet connection with the value of 111 will successfully make the OSPF route more preferred.

Cisco – R2(config)#ip route 0.0.0.0 0.0.0.0 <R1 Next Hop> 111

As a result the R2 routing table will have a quad 0 default route with a next hop of R3. If the link between R2 and R3 ever goes down the default route pointing to R3 will dynamically be taken out of R2’s routing table and the backup ‘Floating’ static route will emerge. Once connectivity and OSPF adjacency come back the OSPF default route will once again take priority.

There are other ways to solve the problem but the floating static route is quick and overall straightforward. Someone asked me what this was once and I didn’t know what they were talking about. For some reason I’d never called this function a ‘Floating Static Route’ even though I’d used and seen it multiple times before. It’s a fitting name.

Fortigate SSL VPN – Redistribution into OSPF

I recently had to setup remote access SSL VPN for a customer on a Fortinet firewall. The network runs OSPF internally and the customer asked we integrate the SSL VPN into their IGP. It’s a simple OSPF implementation that utilizes a VPLS leased line from their provider as a primary route to their COLO and a secondary IPSEC tunnel if the leased VPLS circuit goes down. OSPF handles the automatic failover.

First we went through the steps of configuring the SSL VPN, new IP range, user groups, LDAP settings, public DNS records, etc. After confirming we could successfully connect to the firewall remotely, we had to then setup the firewall to advertise into OSPF the new remote access IP range. Once the new prefix was added to the advertised networks list I began checking the routing table of other OSPF router participants. After all the neighbor adjacency was already in place, but low and behold the new IP range failed to show in the neighboring route tables. After doing a bit of research I discovered that the Fortinet firewall will only advertise the SSL VPN prefix through static route redistribution. The firewall operator must setup the appliance with a blackhole static route to the prefix, and then configure the firewall to redistribute static routes into OSPF.

The main problem with the solution above is that you may not want to redistribute every static route on the appliance into the OSPF domain. The below redistribute button in the UI is an all or nothing option.

A method of filtering static route redistribution on the firewall (like many networking devices) is to use Route Maps. Route Maps are a great way to define routing policy and customize redistribution into varying routing protocols.

First step is to create the Blackhole static route that we will then advertise into our OSPF domain. In the UI go to Network–> Static Routes –> and enter the following (whatever the new remote access IP Range is):

Once the static route’s in place the next step is to create an IP Prefix list. Hop into the appliance CLI and use the below commands.

The first configured rule is to match the SSL VPN IP Range, and the second is to deny all other IP ranges. Once prefix list is done the next task is to create the Route Map with the prefix list reference per below.

The last steps to finish is enabling static route redistribution into OSPF and to apply our new Route Map to the redistribution. The below few commands will get the task completed.

Barring the new prefix is in the network statements of the Fortigate firewall, after all of this you should start seeing the new IP range advertised into the OSPF autonomous system. The above can be done as well with an IP Access List on the Fortigate Firewall, but I find the prefix match list an easier method.

AWS Transit Gateway – Active/Passive IPSEC Tunnels

Like so many others, my organization over the last few years has been leveraging AWS cloud services for IT infrastructure.  It started out with one account, one VPC, one VPN tunnel (2 for redundancy) and a handful of EC2 instances.  Gradually over time this simple cloud presence has multiplied drastically.  There are now numerous EC2 instances, VPCs, VPN tunnels, regions in use, and AWS accounts.  I was not around to witness this AWS footprint grow; I inherited the network responsibilities after the fact. 

When arrived I investigated the on premises to AWS connectivity and discovered the web of data center to VPC VPN tunnel networking.  Setting up internal connectivity from an on prem router/firewall to a VPC is relatively straight forward.  In AWS you follow the steps here to setup a site to site IPSEC tunnel and then download a configuration ‘walk through’ to configure your physical device where you want connectivity.  Each VPN tunnel you configure on the AWS side results in two tunnels to your on prem gateway for redundancy.  In the end on the customer network side you get two VPN tunnels to every VPC needing internal network connectivity.  As one can imagine this doesn’t scale very well.  Even 15 AWS VPCs results in 30 VPN tunnels with 30 separate BGP peers or static routes depending on which option you go for in terms of routing.

I was able to simplify my org’s cloud networking by utilizing AWS’ Transit Gateway service.  A Transit Gateway (TGW) essentially moves your cloud networking architecture to a hub and spoke model.  Instead of setting up direct connections from your data center to each VPC separately, each VPC is connected to the TGW and the local router/firewall has an IPSEC VPN tunnel only connecting to the TGW as well.  The TGW allows for all internal connectivity between VPCs and on prem.  Per the example above, moving to a TGW hub and spoke you would go from 30 VPN tunnels to two.  The two VPN tunnels to the TGW are for redundancy and equal cost multi-path if you want to go the ECMP route.     

    Example Before:

    Example After:

Per the AWS Transit Gateway documentation, the maximum bandwidth per VPN connection is 1.25 Gbps which exceeds the bandwidth of the internet connection the org is using to terminate the site to site tunnel.  This means ECMP is not needed and an active/passive tunnel approach is more suitable.  An issue I ran into while setting this up had to do with tunnel priority and BGP.  I discovered that when setting up a standard VPN tunnel direct to VPC, AWS will advertise routes to an on prem device over BGP with a MED value of 200 for one tunnel and 100 for the other.  This allows for a primary and secondary tunnel, avoiding asymmetric routing, etc.  However, when two tunnels are configured to a TGW AWS does not advertise routes to a customer on prem device with different MED values.  This obviously is an issue if you’re looking for an active/passive approach with zero ECMP.

After digging around AWS documentation and the internet I discovered that BGP peering to a TGW will in fact honor AS Path Prepend.  As a result I was able to use AS Path Prepend on the routes advertised over the secondary tunnel, and Local Preference on the routes advertised to our on prem device from AWS over the primary tunnel.

I believe these same BGP attributes can be used for AWS DirectConnect customers as well.

For some reason I struggle to find proper AWS documentation and setting this up took longer than it needed to.  I hope this will save someone some time if they run into this same situation.