The plastic timing cover on my 2016 Volkswagen Golf R was leaking. Well, not exactly leaking but a slow annoying weeping of motor oil around the cover that hides the intricate mechanical timing components of the 2 liter four-cylinder turbocharged engine.

I tried the usual steps to remedy this common problem area of the Volkswagen Mk7 (and prior) Golf platform: tightening the cover bolts and making sure my positive crankcase ventilation (PCV) system was functioning properly. Alas, oil would still make its way past the seals and throughout my engine bay.

After purchasing a new upper timing cover and seals and spending about 2 hours time on installation, I was able to resolve the issue. In this post I'll be going over everything you need and some helpful tricks in order to resolve this common issue.

Parts Required

ShopDAP has a Upper Timing Cover Reseal kit that will provide all the hardware needed for this job. The total price of this kit is $99 plus tax.

Upper Timing Cover Reseal Kit | Gen 3 2.0T / MK7 & MK7.5

Fix the Oil Leak Coming from the Upper Timing Cover on your 2.0t Engine

Manufacturer:Sign In

The individual part components are as follows:

A new upper timing cover will come with the seals already affixed, which is convenient. Do make sure the seals are seated properly on the cover before installing on your vehicle. You may or may not be able to get the seals themselves without the plastic cover. I did not go this route but it may be viable as long as your cover has not warped.

The two seals for the camshaft magnets are optional, but are a good idea to do while you have the upper timing cover off and have access to them.

Technically, the torx screws are one time use and it is advised to get new ones.

Tools Used

The Harbor Freight ICON Locking Flex-Head Ratchet and Bit Set is key to a successful DIY job. The torx bits on the upper timing cover and two cam magnets are in a tight position as they are very close to the engine mount. This ratchet with its pass-through bit design allows access to these areas without needing to remove the engine mount (which would also require properly supporting the engine).

Another tool that is helpful is a thin 10mm open-ended wrench. I am a fan of this super thin set of wrenches by Capri Tools that comes in 7 different sizes. During my research, some individuals had found success with a long ratching box end wrench. I would advice against this. The two bolts on the bottom come very close to the engine mount as they are backed out. You may get stuck with a box end wrench due to the limited clearance there.

Other tools include: a pick set to remove the cam seals, a set of pliers to remove a coolant hose coming from the expansion tank, and trim tools to pry the cam magnets from the upper timing cover.

Upper Timing Cover Removal and Install

Slide the hose off the metal barb shown in the above picture. You will also need to unbolt the metal barb to give clearance during removal of the cover.

To remove the connectors on the two cam magnets, take a pick and push the clip up. Then press the clip down and pull the connector out.

Next you will need to unbolt the oil dipstick tube to give you more space while removing the cover. You do not need to remove the dipstick itself. You can see the bolt in the above picture.

There is another plastic clip the dipstick tube is connected to outlined in the picture above. With this, the dipstick tube should be able to be moved out of the way.

Now you can remove the six bolts (three each) holding the camshaft magnets. Carefully pry the magnets out. They like to pop out and drop down the engine bay.

This will give you the most room for removing all of the bolts for the upper timing cover. The bottom two bolts are the worst to get to. Use the thin open ended wrench to work the two bolts lose. I could not get my fingers on the bolts to spin the bolts and had to slowly wrench them out. Unfortunately, the Harbor Freight flex-head ratchet will not fit in the bottom space, but does make easy work of the other four bolts that hold the cover on. The bolts are locked onto the cover itself and will not need to be removed completely to remove the cover from the engine block.

With the bolts backed out as far as they will go, carefully remove the cover. This is easier said than done and will require a bit of finesse.

With the cover removed (sorry no pictures of that), be sure to clean the mounting surfaces where the seals rest with a shop towel. Now is the perfect time to replace the two camshaft magnet seals with a pick. Remember to move over your oil cap to the new cover as well.

Installation is the reverse. The bolts like to get in the way while you work the cover into place. Do not try to force anything too hard and be careful of the seals being moved out of place.

Hand tighten and then torque the six cover bolts in a star pattern. The final torque specs for these bolts is 9Nm. Clean and reinstall the two cam magnets and replace the six aluminum torx bolts. These torque to yield bolts must be torqued to 4Nm then turned an additional 45 degrees.

Replace the bolt for the oil dipstick tube, replace the coolant hose, and you are all done!

The Internet has a problem, with an ever-increasing amount of devices persistently connected, we are running out of addresses to provide through the originally developed and widely used Internet Protocol v4 (IPv4).

This has been anticipated since the late 1980s and all of the top-level address spaces have been allocated as of 2011. Do not fret, as there is an increasingly popular standard on the horizon, the Internet Protocol v6 (IPv6), which will vastly expand the number of addresses available. In addition, IPv6 provides some fundamental improvements to the outdated IPv4 architecture, helping to improve the performance and reliability of the routing of data to our precious connected devices.

The king is dead, long live the new digital messiah!

Addressing Architecture

One of the limiting factors of IPv4 is the number of unique addresses that may be allocated to connected devices. IPv4 has a 32-bit address space, allowing the enumeration of around 4.4 billion addresses to be allocated globally.

There are restricted blocks of addresses, used for special purposes such as for private networks, further dwindling the number of addresses that may be used for devices connected to the internet.

In contrast, IPv6 uses a 128-bit address space, allowing for a staggering 340 undecillion (trillion trillion trillion) unique addresses. It is hard to conceptualize the vastness of this span of numbers, but this quote from an article by CNN may help put it into perspective:

With IPv6, there are now enough IP combinations for everyone in the world to have a billion billion IP addresses for every second of their life.

Packet Headers

With a quadruple increase in address size, one might think that IPv6 packet headers, which specify the addresses the data is to be transmitted to and from, may be at least four times as large. In fact, the header size is only twice as large as seen with IPv4 packets.

This is due to changes to the structure of IPv6 packets by removing no longer needed components found in IPv4. These changes have made for more performant packet forwarding by creating as lean of a packet size as possible while supporting the necessary growth of the Internet. Meta reported seeing 10-15% faster page loads due to their transition to IPv6-only deployments in 2015.

Checksum Field Has Been Dropped

The IPv4 checksum field, which was used to verify that the datagram contained within a packet had not been corrupted during transit, has been dropped.

The reasoning behind this change is that devices already perform this action by validating the integrity of the transmitted data at the data link layer (Layer 2). Layer 4 protocols, such as TCP and UDP, are also encouraged to perform redundancy checks. This is done as a part of the default TCP operation. For UDP, checksum validation is optional in IPv4, yet is required to be used under IPv6.

Packet Fragmentation Has Been Dropped

IPv4 handles cases where the Maximum Transmission Unit (MTU), defining the maximum size of a packet, is smaller than the transmitted packet size, by fragmenting a packet along its destination into smaller units. IPv6 does not support fragmentation past the originating source. Routers will discard packets larger than the MTU defined on the interface. Instead, the source of the packet is responsible for fragmenting it using the appropriate size.

Address Types

Address type         Binary prefix        IPv6 notation
------------         -------------        -------------
Unspecified          00...0  (128 bits)   ::/128
Loopback             00...1  (128 bits)   ::1/128
Multicast            11111111             FF00::/8
Link-Local Unicast   1111111010           FE80::/10
Global Unicast       (everything else)

An IPv6 address is represented as eight groups of four hexadecimal digits, separated by colons. This can be shortened by using a double colon, ::, to represent consecutive zeros.

There are several important types of addresses defined by the IPv6 addressing architecture standards that we will learn about below.

Unicast Addresses

A unicast address is an identifier for a single interface. A packet sent to a unicast address is delivered to the interface identified by that address.

Link-local unicast addresses are special uniquely generated addresses defined and used locally to a particular network segment. Every interface supporting IPv6 is required to have a link-local address. These addresses are not guaranteed to be unique and are not meant to be routed across network boundaries.

Global unicast addresses (GUA) are globally unique and externally routable addresses on the IPv6 Internet. As defined by RFC4291, GUAs may be any address excluding the range of FF00::/8. The Internet Assigned Numbers Authority (IANA) has allocated the space 2000::/3 for GUAs.

Anycast Addresses

An anycast address is an identifier for a set of interfaces, belonging to different nodes. A packet sent to an anycast address is delivered to one of the interfaces identified by that address. The interface chosen is considered the nearest, according to the routing protocol's distance measurement.

Use cases for anycast addresses include load balancing and efficiently distributing data among the nodes in a network. As an example, if a router fails, IPv6 nodes may establish a connection to a fail-over router, using a defined anycast address, and maintain the availability of the network.

Anycast addresses are indistinguishable from unicast addresses.

Multicast Addresses

A multicast address is an identifier for a set of interfaces typically belonging to different nodes. A packet sent to a multicast address is delivered to all interfaces identified by that address. Multicast addresses are present in the address space: FF00::/8.

There are no longer IPv4 broadcast addresses in IPv6. A broadcast address is used to transmit network information, such as DCHP and ARP, to all hosts on a network segment. IPv6 subnetworks are designed to be substantially larger and broadcast traffic becomes in-efficient at this scale. Their function has been replaced by multicast addresses.

Well-Known Multicast Addresses

There exists an address registry, managed by IANA, covering well-defined multicast addresses for standardized topics such as:

• The all routers group (FF01::2)
• The all nodes group (FF01::1)
• The Multicast DNS (mDNSv6) group (FF01::FB)

Multicast Listener Discover

Multicast Listener Discover (MLD) is used to join and listen to multicast traffic. Devices send MLD reports on groups that they would like to join. Routers receive the reports and build multicast tables associated with the addresses. When the router receives multicast traffic, it is responsible for forwarding to the individual interfaces.

Thread - A Use Case for IPv6 Address Types

We are seeing applications and standards utilizing these different types of addresses for optimizing the flow of traffic. For example, Thread, an open IPv6-based networking protocol for Internet of Things (IoT) devices, uses these address types for the operational communication of connected smart devices.

• Multicast messages define and allow control of connected groups of nodes and allow devices to announce themselves on the network for service discovery.
• Anycast messages determine the closest router to a particular node in a Thread mesh network. In addition, anycast addresses allow fault tolerance. If a router becomes unavailable, a new router is established and connectivity is maintained.

Subnetworks in IPv6

The recommended size of an IPv6 subnetwork is 64 bits. This allows for 2⁶⁴ individually addressable hosts within the subnet. This is substantially larger than IPv4 and in fact, a single IPv6 subnetwork can contain the entire address space of IPv4.

The vastness and consistent size of IPv6 subnetworks have many benefits including:

• They support the future growth of IoT devices by significantly increasing the number of local connections available.
• They make it easier for network engineers to manage and move subnetworks.
  • There will be a less likely chance of host address collisions when connecting large organizational networks.
• Being less feasible for attackers to use network probing techniques that sequentially scan to find open services for attack, as talked about in RFC 5157.

Network Prefix Sizes

A "Prefix" is a defined range of blocks in the IPv6 address space beginning with the left-most bits. As an example, 2001:0db8:1234::/48 indicates that the first 48 bits, 2001:0db8:1234, are for the global routing prefix. The remaining 80 bits can be divided by the assigned part to delegate to routers for subnetworks and host addressing.

Internet Service Providers (ISPs) are recommended to provide an IPv6 prefix of at least /56 to homeowners and a /48 for organizations. Given these prefix sizes and the recommended IPv6 subnetwork size of 64 bits, we can calculate the total subnets available to each.

• A homeowner receiving a /56 prefix could create 2⁸ individual subnets.
• An organization receiving a /48 prefix could create 2¹⁶ individual subnets.

64-bit Extended Unique Identifier (EUI-64)

The interface identifier (IID) is the grouping of the right-most 64 bits of an IPv6 address. It is not meant to be contiguous but instead, each client generates their unique identifier.

You may be aware of IEEE 802 MAC addresses, which are globally unique identifiers assigned to all commercial networking hardware. These identifiers have been used since the early adoption of Ethernet and standardized as part of IEEE 802.3. While these 48-bit addresses allow for a large number of unique identifiers, the standard is projected to have an end-of-life by 2080.

IPv6 has adopted the EUI-64 standard, which provides an increased 64-bit capacity and is derived from a MAC address using a straightforward process. We will look at alternatives to generating 64-bit IPv6 interface identifiers to preserve privacy in the sections below.

Dynamic Host Configuration Protocol (DHCP) Version 6

The fundamentals around DHCPv6 are different when compared with its DHCPv4 predecessor. The biggest change is that a DHCPv6 service is no longer required to be run on each network segment to coordinate host addressing. A DHCPv4 service was required to be present and be the source of truth for host address reservations in IPv4. On the other hand with IPv6, clients can generate addresses from self-assigned identifiers using what is known as Stateless Address Autoconfiguration (SLAAC).

The primary responsibility of DHCPv6 is to provide a mechanism for delegating IPv6 prefixes to other routers, as defined by DHCPv6 Prefix Delegation (DHCPv6-PD). A requesting router acts as a DHCP client and requests a prefix to be assigned by a delegating router, acting as a DHCP server.

As part of recommendations for DHCP-PD, delegated prefixes should have an indefinite lifespan. It can be costly to renumber an entire site when provided a new prefix, as each router must reconfigure their DHCP prefix settings and hosts must reconfigure their addresses. ISPs are encouraged to assign customers persistent prefix assignments which would not require renumbering.

Stateless Address Autoconfiguration (SLAAC)

SLAAC is often used as a means for obtaining IPv6 addresses, especially for link-local. The following steps are executed by a device to set up an IPv6 stack using SLAAC:

1. The device generates an EUI-64 and with it, a link-local address.
2. The device executes Duplicate Address Detection on the network segment. If a duplicate address exists, the device will restart the process.
3. A Router Solicitation (RS) is sent and a Router Advertisement (RA) is received to obtain the router prefix information.
4. The device creates a Global Unicast Address (GUA) using the prefix provided by the router by executing steps 1 and 2 with this new address.

At this point, the device can communicate with neighbors on the network segment and globally through the IPv6 router advertised.

SLAAC allows for simplified network configuration as stateful services, like DHCP, are no longer necessary to run an operable network. Devices can automatically configure their addresses and communication can take place throughout the networking segment.

Advertising DNS Configuration

RFC 8106: IPv6 Router Advertisement Options for DNS Configuration

This document specifies IPv6 Router Advertisement (RA) options (called “DNS RA options”) to allow IPv6 routers to advertise a list of DNS Recursive Server Addresses and a DNS Search List to IPv6 hosts. This document, which obsoletes RFC 6106, defines a higher default value of the lifetime of the DNS RA options to reduce the likelihood of expiry of the options on links with a relatively high rate of packet loss.

IETF DatatrackerJaehoon Paul Jeong

To support IPv6 SLAAC deployments without requiring a DHCPv6 server running on each subnetwork, a mechanism was introduced to provide DNS configuration as part of the Router Advertisement (RA). Domain information, such as RDNSS and DNSSL, may be included in the RA messages to inform hosts of this critical domain information.

End-to-End Routing

Due to the anticipated depletion of IPv4 addresses, Network address translation (NAT) was introduced. This routing technique allows a single public IPv4 address to map to multiple local devices on a private network. When a local device connects to an external resource, a router must keep track of the connection state and context, and translate incoming and outgoing communication between the devices.

With IPv6, NAT is no longer required. Every single connected device can receive a unique and publicly accessible IPv6 address. Thus connections are truly end-to-end and no translation by the router is needed. In its purest form, every host can directly communicate with another host through its IPv6 address, barring any firewall restrictions between the two hosts.

Neighbor Discovery Protocol (NDP)

The Neighbor Discovery Protocol (NDP) is fundamental in IPv6 routing and discovery processes. It is used to perform several important tasks such as:

• Router discovery to solicit and advertise routers on a network segment.
• Address resolution to locate neighboring hosts on a network segment, for Duplicate Address Detection and IP to MAC address translations.

Similar to IPv4 ARP, NDP is used to translate IP addresses with some added benefits that can prevent the security shortcomings of ARP. For example, ARP spoofing can be mitigated by what is called Secure Neighbor Discovery (SEND).

IPv6 Security Guidelines

Security was a fundamental design consideration of IPv6. In fact, IPSec is considered a node requirement for all IPv6 implementations, though it must be enabled on both device and application to use.

Some additional security considerations should be noted and may require extra configuration on network and end devices which we will go over below.

A Firewall is Required

Without Network Address Translation (NAT), IPv6 hosts are directly addressable to hosts outside the network. This fact makes packet switching and routing much simpler and as the Internet had originally been designed for, by way of the end-to-end principle.

That being said, this may cause unintended side effects as hosts on a network designed in a way that assumes hosts would not be publicly addressable may find their hosts exposed directly to the Internet, bad actors, and more when switching to IPv6.

For these reasons, a firewall must be enabled on an IPv6-enabled router and should be configured in such a way that blocks unintended incoming traffic to its hosts.

Increased Security Through Privacy Addresses

The IPv6 addressing architecture defined that the rightmost 64 bits, associated with the interface identifiers (IID), be derived using an interface's MAC address, defined as modified EUI-64 format. This poses security concerns as a MAC address is globally unique and could be used to track an individual. Another security concern is that this could allow an attacker to find vulnerable targets, as MAC addresses can identify specific types of manufactured hardware.

One simple way around this is to generate a random MAC address. The Android operating system implements MAC randomization for each network it connects to. The randomized MAC address can be used to generate an EUI-64 without privacy concerns.

Another way to enhance device privacy is to use Privacy Extensions for SLAAC. Privacy extensions define a mechanism for devices to create temporary addresses using randomized interface identifiers. These addresses are only used for outgoing connections and have a limited lifetime and are rotated periodically, usually 24 hours.

The currently recommended way of creating stable IPv6 interface identifiers, while preserving the privacy of devices, is to use Stable Privacy Addresses defined in RFC 7217. These addresses are created from a hashing function using inputs such as a device generated secret key and the network prefix. This type of stable address remains the same unless it moves to a new network or the network prefix changes.

With all of these different methods of creating the 64-bit interface identifier, the bits representing the identifier lose meaning and so it is recommended that the identifier be considered opaque.

Unauthorized Link-local Addressing

IPv6 traffic may be happening between hosts through link-local addressing even if a network admin has not explicitly set up IPv6 on their network. IPv6 and link-local addresses are auto-configured by the hosts themselves.

One consequence of this is that there may be unintended communication between hosts on a network segment if they are IPv6 enabled. IPv6 and IPv4 addresses should always be considered when defining firewall rules and network Access Control Lists (ACLs).

VPN Leakage for Dual-Stack Environments

A client that supports both IPv4 and IPv6 and relies on VPN connectivity to maintain anonymity, should always make sure that IPv6 is enabled on the VPN client. If the VPN tunnel lacks support for IPv6, the client may inadvertently leak IPv6 traffic outside of the tunnel breaking the user's anonymity.

Virtual Private Network (VPN) traffic leakages in dual-stack hosts/ networks

The subtle way in which the IPv6 and IPv4 protocols co-exist in typical networks, together with the lack of proper IPv6 support in popular Virtual Private Network (VPN) products, may inadvertently result in VPN traffic leaks. That is, traffic meant to be transferred over a VPN connection may leak out of such connection and be transferred in the clear on the local network. This document discusses some scenarios in which such VPN leakages may occur, either as a side effect of enabling IPv6 on a local network, or as a result of a deliberate attack from a local attacker. Additionally, it discusses possible mitigations for the aforementioned issue.

IETF DatatrackerFernando Gont

If the network the client is connected to does not support IPv6, one should still enable IPv6 on the VPN client and black hole all IPv6 traffic. This is done by ProtonVPN as a preventative measure against leakage.

Unique Local Addressing (ULA) and NAT

Network admins may consider, when transitioning from IPv4 to IPv6, keeping the fundamentals of NAT in place within their private networks. Unique Local Addresses (ULA) were defined in IPv6 standards as a block of addresses accessible site-wide and not meant to be routed to the global address space. This would allow NAT66 to map a Global Unicast Address to an internal ULA address.

As explained in this Infoblox article, usage of ULA is fundamentally flawed within a dual-stack environment since IPv4 will always take precedence over ULA addresses. So the IPv6 stack would never be utilized over the IPv4 counterpart.

In conclusion, it is advised to design an IPv6 network around Global Unicast Addresses and not continue with the necessary designs and mindsets commonly used due to the limiting factors of IPv4.

Rogue IPv6 Router Advertisements

Routers on an IPv6 network advertise information to nodes that enable them to auto-configure and connect to their network. Rogue IPv6 router advertisements may be sent to nodes, whether maliciously or by misconfiguration, which could lead to Man-in-the-Middle attacks or general network routing issues.

A similar situation can happen in IPv4, where rogue DHCP servers may be on the network. DHCP snooping can be deployed on layer 2 switches to help combat and secure networks for this IPv4 scenario. An equivalent mechanism, called RA-Guard, can be used for IPv6-enabled routers on layer 2 switches to help detect and mitigate rogue routers on the network.

Neighbor Discovery Protocol (NDP) Attacks

Duplicate Address Detection (DAD) is integral to the functioning of SLAAC. DAD may be exploited by malicious actors for denial-of-service attacks on a network segment. When a node queries with a Neighbor Solicitation to determine whether an address is being used, the malicious attacker may repeatedly respond causing the node to continually reject addresses and be unable to connect to the network. Similarly, an attacker may try to flood routers with NDP queries, causing legitimate traffic to be disregarded or the router's Neighbor Cache to be overloaded.

Secure Neighbor Discovery (SEND) can be used to mitigate these types of attacks. SEND must be enabled on each device, which includes steps like creating public keys and associating Cryptographically Generated Addresses (CGAs). I have not found any operating systems that support this out of the box and only a few third-party concepts are available for use at the time of writing. Bootstrapping each device to mitigate these types of attacks would be very difficult as well.

Alternatively, routers may rate limit the amount of NDP queries that can be sent over a particular time frame.

It should be noted that IPv4 has the same types of attack vectors through ARP. This Infoblox blog article goes over steps an admin can do to secure both protocols but concludes that this is an avenue that requires more thought and consideration to secure.

The Adoption of IPv6

We have seen that there are benefits to switching to IPv6, such as the increase in address space, simplified networking configuration by not requiring a running DHCP server for each subnetwork, and enhanced support for multicast and anycast traffic that allow for innovations in IoT systems.

That being said, the adoption of IPv6 has been slow. Standards for IPv6 were first introduced in 1995. According to the most recent adoption by country statistics reported by Akamai (as of June 2024), only a handful of countries have greater than 50% adoption rates, with the majority of countries having significantly lower rates.

The reasons for the slow adoption are complex. Resources on the Internet will want to be able to reach the most amount of users possible. The current Internet is dominated by IPv4 users, so it makes sense that services will prioritize IPv4 to maximize their reach. In addition, organizations and customers require that their cloud providers or ISPs provide them with sufficient prefixes and support for IPv6 utilization. On top of that, hardware vendors and networking software must be IPv6 aware to support it at every hop.

Some recently passed initiatives will accelerate the adoption of IPv6. The US Government has put policies in place to completely transition to IPv6-only deployments by 2025. It is recognized that IPv6 is the future and the government wants to fully transition its resources to IPv6 and minimize the impact of running dual-stack environments.

As the number of addresses in the IPv4 space dwindles, the cost of owning and allocating them is increasing. As of February 1st, 2024, AWS has started to charge tenants per public IPv4 address allocated to their resources. Supply and demand will dictate the price for these addresses. Organizations will begin to see higher operational costs for IPv4 which may push them to begin transitioning their environments to IPv6.

Conclusion

We have covered many areas around the IPv6 architecture and looked into differences between it and its predecessor, IPv4. In doing so, realizing that it is straightforward and allows for true end-to-end networking, as the Internet had originally intended. We have looked into use cases for the different address types and how they can make for more efficient and reliable routing of our data. Finally, we've looked into how to secure deployed IPv6 networks.

As IPv6 becomes increasingly adopted, we will see some more benefits and greater innovations further fueling the push to switch to the newest protocol. I for one am getting up on the bandwagon to welcome our new digital messiah!

In this post, we will take a look at some easy to install devices, such as the Shelly 1. The device can be used to engage a typical garage door opener, sense whether the door is open or closed, and react to events within our garage. We will use the Home Assistant open source software to manage the devices and create a few automations we need to make our garage truly smart!

Making a Garage Door Opener Smart with Shelly 1

The Shelly 1 is a small yet very capable WI-FI operated relay switch. With this device, we can remotely trigger our garage door opener to open or close the door. In addition, we can wire in a contact switch in order to remotely detect the positioning of the door.

Powering the Shelly 1

The Shelly 1 is capable of being powered by a 12V or 24-60V DC power supply. You should choose a power supply that is within the maximum voltage capabilities of your contact switch. In my case, my contact switch supports up to 28V DC. I had an unused 12VDC power supply lying around, so I repurposed it to power my Shelly 1 and contact switch.

The positive wire from the DC power supply should go to the N+ terminal, while the negative wire goes into L-. Before turning on the device, make sure to set the jumper properly on the internal Shelly 1 contacts, which are under the removable cover. The Shelly 1 that I received was pre-configured for AC and not DC.

See the official wiring diagram for more details on how to wire the device.

Configuring the Shelly 1

When powering on the Shelly 1 for the first time, the devices creates a WI-FI access point to connect to. You should see the WI-FI network as "shelly1-XXX". Connect to this access point using a laptop or mobile device, and navigate to 192.168.33.1 to access the web UI to configure the device.

First, go to Internet & Security and will out the options under WI-FI Mode Client to connect to your homes WI-FI network.

Under Timers, we want to configure the Auto Off setting. Set a value of 0.5 seconds. What this will do is when we have fully wired the device and it is remotely activated, the internal relay will close the circuit to the garage door opener. This triggers the opener to open or close. The Shelly 1 will automatically open the circuit after half a second, which will prevent the opener from continuously opening and closing the door.

Under Settings, set the Power On Default Mode to Off. And for Button Type, select Detached Switch. This will prevent changes to the state of the garage door sensor from triggering our garage door opener.

You may consider checking the Reverse inputs option as well. If your switch is the NC (Normally Closed) contact type, the Shelly 1 firmware will consider the switch "open" when the door is closed. Reversing the inputs will display "closed" in this case.

For more information, see the Shelly 1 web interface guide.

Connecting the Shelly 1 to the Garage Door Opener

Connecting the Shelly 1 to the garage door opener is very simple. On the side of the opener, there are usually several terminals. Refer to your manual on exactly how these terminals are used to initiate opening and closing of the garage door. This is the description in my manual for manually testing the opener:

Connect the opener to a power source and short across the screw terminals labeled "PB" and "COM"...

So, I connect the COM terminal on the garage door opener to the I terminal on the Shelly 1 (the I terminal represents the input load) and connect the PB terminal to the O terminal (which represents the output load).

Once connected, you should be able to go into the Shelly 1 web interface and trigger the door to open and close!

Determining the Position of the Garage Door

In order to determine whether the garage door is open or closed, we wire in a magnetic contact switch. This is also known as a reed switch and uses a detached magnet. For NC contact switches, when the magnet is near, the circuit will close allowing current to flow. When the magnet is moved away, the switch will open and the current will not flow. The Shelly 1 can detect the flow of current through the switch to determine the positioning of the door.

I found a convenient spot above my garage door, where I was able to screw in the wired switch. Then, I mounted the magnet portion on to the door itself in close proximity to the switch.

When the door is closed, the magnet is near and the circuit is complete. When the door is opened, the magnet moves away from the switch and the circuit is opened. We can not know how much the door is open. But this will give us a good indication whether the door is closed or not. In my testing, since the door opens both up and away from the wall, the magnet moves away from the switch rather quickly.

I ran the wires to the switch along the outside of the metal frame for the chain. My Shelly 1 is placed right above the door opener.

Wire in the switch to the SW terminal of the Shelly 1. And place the other wire for the switch into the L- terminal.

We will be able to tell whether the switch is working properly by going into the Shelly web interface. When the door is open, the middle of the button illuminates blue. Otherwise, this portion will be white.

Integration with Home Assistant

At this point, we have a pretty functional way of opening and closing the garage door as well as seeing whether the door is closed or not. We can go further by using the Shelly integration for Home Assistant.

After configuring the integration, we will have access to two entities in Home Assistant. One switch, named "switch.shelly1_XXX", will allow us to engage the garage door opener to open or close. The other binary sensor, "binary_sensor.shelly1_XXX", will allow us to sense the positioning of the door.

At this point, we can create a nice dashboard with our entities. The above is an example that I have configured in Home Assistant.

Alert! Garage Door is Open!

Have you ever come home to realize that you left your garage door open while you were away? Luckily, we are now able to use Home Assistant to alert us when the door has been left opened for a certain period of time.

Before we do this, we need to edit our Home Assistant, configuration.yaml to specify a notify integration. There are many different options for sending notifications. I choose to use SMTP, since I can send an email as well as a text message to my AT&T number using this method.

Here is an example for using Gmail. You may create an App Password to use with the configuration below.

notify:
  - name: email_notification
    platform: smtp
    server: smtp.gmail.com
    port: 587
    timeout: 15
    sender: <replace>@gmail.com
    sender_name: "Home Assistant"
    recipient: <replace>@gmail.com
    starttls: true
    username: <replace>@gmail.com
    password: <replace>

Go to the Settings -> Automations and create a new automation. You can switch to Edit in YAML in order to copy the following (replacing the entity_id with your corresponding Shelly 1 entity):

alias: Garage open notification
trigger:
  - platform: state
    entity_id:
      - binary_sensor.shelly1_XXX_input
    from: "off"
    to: "on"
    for:
      hours: 0
      minutes: 10
      seconds: 0
action:
  - service: notify.email_notification
    data:
      message: Garage door is open!
      title: Garage door is open!
      target: 111111111@mms.att.net
  - service: notify.email_notification
    data:
      message: Garage door is open!
      title: Garage door is open!
      target: some-email@example.com

This will send an email and a text message after the garage door has been opened for more than 10 minutes.

Many mobile networks will allow emailing to a cell phone number through a specific carrier address. For example, AT&T will allow sending an email as a text message using the address structure, <att-number>@mms.att.net.

If you receive this message, a good idea would be to have your smart phone configured so that you may remotely connect into your network. In order to access your Home Assistant remotely. Take a look at the following post for instructions on how to set that up with pfSense.

Remote VPN Server with pfSense and a Dynamic IP Address

How to configure a pfSense router for remote access using OpenVPN. These instructions will target residents who have a dynamic IP address.

The Generally Available BlogPatrick Jennings

Turning On Lights When the Garage is Opened

I use a Kasa motion sensor light switch in my garage to activate and control my overhead LED lights. This light switch is also connected to Home Assistant. It activates well when motion is sensed in the garage, but it is much nicer to have the lights turn on as soon as the garage is opened instead of waiting for detected motion.

We can do this easily enough with the following Home Assistant automation.

alias: Garage Lights on when Garage opened
trigger:
  - platform: state
    entity_id:
      - binary_sensor.shelly1_XXX_input
    from: "off"
    to: "on"
action:
  - entity_id: switch.garage_lights
    service: switch.turn_on

NFC Tags for Opening the Garage

Home Assistant can program NFC Tags, which can then be scanned and used for automations. We can use this to place a tag outside, which can then be scanned by a smartphone and trigger the garage door opener.

In Home Assistant, go to Settings -> Tags. Create a new Tag. This will create a unique tag ID that only our Home Assistant installation will have any association with. What is being written to the NFC tag, will be a URL in the form of https://www.home-assistant.io/tag/<tag id>. Only a device authenticated and connected to our Home Assistant can trigger this automation.

We can associate this tag with a new automation. That will trigger the garage door to open or close when the tag is scanned:

alias: Open Garage when Garage Door Tag is scanned
trigger:
  - platform: tag
    tag_id: <tag-id>
action:
  - service: switch.toggle
    target:
      entity_id: switch.shelly1_XXX

With a mobile phone supporting NFC capabilities and using an official Home Assistant companion app, we can write this tag to a programmable NFC tag.

As a side note, if you will be adhering this tag to a surface, you should first test that the tag scans properly beforehand against that surface. I have found that the tags will not scan as well, or even at all, when placed near metal objects, which may be near or behind the wall or surface the tags are placed on.

And that is it! Scanning the NFC tag will trigger the door to open or close.

Conclusion

With a few connected IOT devices, a little bit of wiring, and some software configuration, we are able to make our garage smart and connected. This will enhance the safety of our home in addition to making it more convenient to operate. Finally, none of these devices require a connection outside of our home network or to external services, so we are preserving our privacy as well.

Products

If you are interested in the products that I used, here are some affiliated links to checkout.

The OAuth framework has gained significant traction in our industry. Making it the leading method of authorization. OAuth provides groundwork for authorizing end-users as well as non-human services through the use of different grant types. With such a flexible and widely used authoritative framework, it is important to keep up to date with the current best practices in order to keep our users and services secure.

This post will follow the guidelines and best practices detailed in the Internet Engineering Task Force article: OAuth 2.0 Security Best Current Practice.

In addition, we will cover these topics as it applies to the open source IAM software, Keycloak. You should have some basic knowledge of Keycloak, such as creating realms, as well as have some basic understanding of OAuth.

How May OAuth be Attacked?

OAuth implementations are being attacked through a few weaknesses and anti-patterns.

This includes the usage of some of the legacy grant types, such as implicit grants and password grants. As well as bypassing intrinsic security defenses in place, like with so called "Open Redirectors". Finally, the infrastructure and configuration of the authorization server (i.e. Keycloak) side may allow leakage of state, authorization codes, or at worst access tokens.

OAuth is being used in environments requiring higher security standards. Such as in the banking industry as well as in Government and other public services. Understanding the different types of grants and how to configure them to be as secure as possible is critical to these industries.

We will look at mitigating these known methods of attack below.

OAuth Metadata

The first recommendation is to publish OAuth Metadata, such as defined in RFC8414. And that clients make use of this metadata to configure themselves instead of relying on static configurations.

Keycloak does this automatically by exposing the OAuth metadata in a standard location:

• https://<keycloak-host>/realms/<realm>/.well-known/openid-configuration

This includes important fields and authorization server capabilities, such as the authorization_endpoint and token_endpoint. In addition, the jwks_uri, which links to a URI which lists the public keys used by Keycloak to sign JWTs.

To learn more about the use of this metadata and connecting client libraries, see the Keycloak documentation below:

https://www.keycloak.org/docs/latest/securing_apps/#endpoints

Implicit Grants

With Implicit grants, the access token is passed through in the authorization response directly. This means that the tokens would be passed in through the redirection URLs directly. Exposing greater attack surfaces and other attack vectors such as:

• An access token may be saved in browser history.
• If an attacker is able to redirect to a URI under their control (such as through an “open redirector”), they are able to get direct access to the access token.

Thus, this grant type should not be used. And instead authorization code grant type should be used instead.

Resource Owner Password Credentials Grant

The Resource Owner Password Credentials (ROPC) grants, also known as Direct access grants in Keycloak, is another legacy grant type that should not be used in production environments.

This legacy grant type was originally intended to be used to migrate existing legacy applications and services, that were already handling user credentials directly, to the OAuth framework.

ROPC has numerous security problems, including:

• Insecurly exposes credentials of the resource owner to the client.
• Users are trained to enter their credentials in places other than the authorization server.
• Causes problems when trying to implement two-factor auth, WebAuthn, WebCrypto, or any other multi-step authentication process.

Authorization Code Grant

Authorization Code grants are initiated by browser based applications, generally by human or human-like user agents. The user agent is redirected to the authorization server, where they will be authenticated and authorized, and redirected back to the application with an "authorization code". Which can be exchanged with the authorization server for an identity and access token.

In Keycloak, we can create a new client with "Standard flow" checked in order to enable this grant type.

Wildcard Redirect URIs

Registering redirect URIs is a fundamental validation layer with the authorization code grant. The authorization server will detect that a valid redirect URI is passed in when performing this auth flow.

OAuth is being used in much more dynamic environments. At inception, OAuth had actually assumed more static relationships between the client, authorization server, and resource servers.

Wildcard redirect URI patterns emerged as a means of allowing more dynamic client configurations. Unfortunately, these ambiguous patterns pose a problem due to the more complex implementation and are more error prone to manage.

Let's look at the following redirection URI in Keycloak:

This * suffix allows any URI prefixed with "https://somesite.example/" to be allowed to redirect to after a successful login. Valid URIs that match this rule include:

• Any URL path, such as https://somesite.example/apis/users.
• Any query parameters, such as https://somesite.example?utm_campaign=blah.
• Any fragments in the URL, such as https://somesite.example#title

If one is not familiar with how a particular OAuth implementation validates these redirect URI patterns, this may open the clients to exploitation.

For example, let's assume someone is wanting to allow redirects to a particular host and to any port on that host. They might incorrectly consider the following URI pattern:

• https://somesite.example*

What they might not realize is that this may open them up to redirects to a host under an attackers control, such as: "https://somesite.example.attacker".

In general, the IETF article states that one should prefer to use static redirect URIs, without any wildcard, because of issues outline above.

There is one exception. For port numbers in localhost redirect URIs for native apps. In Keycloak, you may use the special redirect URI "http://127.0.0.1". Which will also allow a redirect uri to localhost on any port.

Keycloak is looking to improve this user experience by the introduction of specific client policies. So there would be a client policy that would enable, for instance, wildcard matching only on subdomains or matching only on URL fragments.

You can see the ongoing discussion in the community below.

Clients policies for wildcards in redirect-uris · keycloak/keycloak · Discussion #9278

Keycloak currently allows wildcards in the redirect-uri by default, and allows any scheme to be used in redirect URLs as well. Some examples of valid redirect-uris that can be configured for client...

GitHubkeycloak

Open Redirectors

An open redirector is a specific endpoint that forwards a users browser to any arbitrary URI. For example, a URI may be put in a query parameter in which is used as a forwarding redirection URI to the authorization server:

• https://some.example?redirect_to=https%3A%2F%2Fother.example

Open redirectors allow an attacker to construct URIs pointing to resources that they may control. Exposing the authorization code, in the case when authorization code grant is used, or the access token, if the legacy implicit grant is used.

Proof Key for Code Exchange

It is advised for public clients using authorization code flow to use Proof Key for Code Exchange (PKCE).

Although PKCE was designed as a mechanism to protect native apps, this advice applies to all kinds of OAuth clients, including web applications. For confidential clients (where the "client authentication" option is enabled in Keycloak), the use of PKCE is also recommended. As it may prevent CSRF and mitigate authorization code interception attacks.

PKCE can be enforced, and configured to prevent PKCE downgrade attacks, for a client in Keycloak by enabling a "Proof Key for Code Exchange Code Challenge Method" under the "Advanced Settings". The use of PKCE mode "S256" as the code challenge method is highly encourage.

Client Credentials Grants

Client credentials grants are often used to interconnect services. And are generally authorized using a client id and client secret.

In Keycloak, these are often considered Service Accounts. And enabling this grant type can be done by checking the "Client authentication" (which generates a confidential client secret), as well as checking the "Service accounts roles" during client creation.

Recommendations for Client Credentials

By default, Keycloak generates a random client secret that is sufficiently complicated. The IETF article recommended to use asymmetric methods, such as:

• mTLS
• Signed JWTs

Keycloak supports both of these types of client authentication. Certificates for signed JWTs can be specified by importing, generating, or queried by defining a JWKS URL, in the "Keys" tab.

X509 Certificates may be configured by following:

• https://www.keycloak.org/docs/latest/server_admin/#_x509

Preventing Impersonation of Resource Owners

Resource servers may make access control decisions based on the identity provided by the authorization server.

It may be possible to impersonate a particular resource owner as a service account. If for example, client id were to be able to be specified by the client. It may be possible to impersonate a particular resource user by a certain service account.

The administrators of an authorization server should not allow clients to influence their client id or any claim that could cause confusion with a genuine resource owner.

If this cannot be avoided, authorization servers must provide other means for the resource server to distinguish between the two types of access tokens.

One may configure service accounts in Keycloak to have any client id they wish. It is up to the administrator to establish a robust client id generation schema that prevents this type of impersonation.

Security of Refresh Tokens

The purpose of a refresh token is to allow the access token to be short lived. And allow for requesting of new access tokens, once they have expired.

As advised by the IETF article, issuing a refresh token is optional and should be done at the discretion of the authorization server. In addition, the following conditions should be followed:

The authorization server MAY issue a new refresh token, in which case the client MUST discard the old refresh token and replace it with the new refresh token.  The authorization server MAY revoke the old refresh token after issuing a new refresh token to the client.  If a new refresh token is issued, the refresh token scope MUST be identical to that of the refresh token included by the client in the request.

If refresh tokens are issued, those refresh tokens must be bound to the same scope and resource servers (see below regarding audience restricted access tokens) as originally consented by the resource owner.

Refresh tokens for public clients must be either sender-constrained or use refresh token rotation, as described below, in order to detect compromise:

The authorization server issues a new refresh token with every access token refresh response. The previous refresh token is invalidated but information about the relationship is retained by the authorization server. If a refresh token is compromised and subsequently used by both the attacker and the legitimate client, one of them will present an invalidated refresh token, which will inform the authorization server of the breach. The authorization server cannot determine which party submitted the invalid refresh token, but it will revoke the active refresh token. This stops the attack at the cost of forcing the legitimate client to obtain a fresh authorization grant.

Access Tokens

Access Token Privilege Restrictions

Access tokens represent the authorized privileges and accesses granted on behalf of a user. And as such, must be kept confidential and as short lived as possible.

According to the current best practices, the privileges associated with an access token should be restricted to the minimum required for the particular application or use case. In addition, the resource servers should validate that a given access token is meant for a particular means requested. This should be done through the following:

• Through audience-restricted access tokens. And/or through the use of client scopes.
• Or through authorization_details of RFC9396. Which defines a structured mechanism for specifying fine-grained authorization requirements.

Authorization Request and Access Token Leakage

Contents of authorization request or response URIs may be leaked unintentionally through the Referer header. Suppression of the Referer header should be done by applying an appropriate Referrer Policy. Keycloak will automatically set the policy to no-referrer in order to prevent this type of leakage.

An authorization code may end up in a user's browser history. Attackers may learn state from the authorization server if it contains links or third-party content. Or a client may leak the authorization response, if it includes third-party content.

To mitigate the effects of a leaked token, sender constrained access tokens and audience-restriction should be enforced.

Sender-Constrained Access Tokens

A sender-constrained access token provides methods to prevent misuse of leaked access tokens.

A sender-constrained access token scopes the applicability of an access token to a certain sender. This sender must prove that they were the original recipient of the token for the acceptance of that token at the resource server.

The resource server is the one to preform proof of possession check. This can be done using mutual-TLS client authentication and certificate-bound access tokens.

Audience Restricted Access Tokens

The audience may be associated with a particular access token. In order to restrict the token to ideally one or maybe more resource servers. Audience restrictions limit the impact of token leakage.

In deployments where the authorization server knows the URLs of all resource servers, the authorization server may just refuse to issue access tokens for unknown resource server URLs.

Audience restrictions have benefits beyond token leakage mitigation. But allows the authorization server to create access tokens with differing claims based on the resource server specified.

Keycloak clients can be configured to provide audience through mappers. See the documentation around Audience Support for more information.

Authorization Server Infrastructure

It is recommended to use end-to-end TLS. Which means complete encryption between client, authorization server, and any reverse proxy server in-between.

A reverse proxy must sanitize any inbound requests to ensure the authenticity and integrity of all header values relevant for the security of the authorization servers. For example, the X-Forwarded-For header may be used to indicate the address of a connecting client.

A list of headers that should be sanitizing, when operating Keycloak in one of its proxy modes, can be found below:

• https://www.keycloak.org/server/reverseproxy

Cross-Origin Resource Sharing (CORS) may be enabled on the following endpoints:

• Token endpoint
• Authorization server metadata endpoint
• jwks_uri endpoint
• Dynamic client registration endpoint

CORS should not be used on the authorization endpoint. As the client never accesses this endpoint directly, and only ever redirects users to it.

Keycloak allows one to specify any number of CORS origins through the "Web origins" client configuration.

And there is open discussion for customization of the Access-Control-Allow-Headers returned by preflight requests in the issue below:

[CORS] Allow Access-Control-Allow-Headers customization · Issue #12682 · keycloak/keycloak

Description Allow customization for CORS Access-Control-Allow-Headers Discussion https://keycloak.discourse.group/t/customizing-access-control-allow-headers/7672 Motivation Keycloak has to support ...

GitHubkeycloak

Cross Site Request Forgery

The IETF article states:

Clients must prevent Cross-Site Request Forgery (CSRF). CSRF refers to requests to the redirection endpoint that do not originate at the authorization server, but a malicious third party.

A client may rely on the CSRF protection provided by PKCE. In which case, PKCE must be enforced and not be able to downgrade (see above for directions on configuring that on a client level in Keycloak).

Keycloak prevents CSRF attacks against the login and user account portions, using the state parameter, as described in the documentation:

• https://www.keycloak.org/docs/latest/server_admin/#csrf-attacks

Clickjacking

An attacker may embed the authorization endpoint user interface in an innocuous context, such as an iframe.

IETF article states that the authorization servers must prevent clickjacking attacks.

• By specifying the X-Frame-Options HTTP response header.
• By utilizing Content Security Policy (CSP) level 2 [CSP-2] or greater.
•  Older browsers do not all support the required CSP levels to prevent iframe clickjacking. So JavaScript based framebusting techniques should be used.
• Authorization servers should allow for configuring specific allowed origins for particular clients.

Keycloak prevents Clickjacking by setting CSP and X-Frame-Options headers as described in:

• https://www.keycloak.org/docs/latest/server_admin/#clickjacking

Conclusion

As we have seen, the OAuth framework provides a lot of types of grants for the many use cases applications need in order to authorize and authenticate users and services.  Unfortunately, there are a lot of configuration options which may allow for insecure methods of exchange between user agent, resource server, or authorization server. But we have at IETF's article regarding OAuth 2.0 best practices. And have applied some of this knowledge to prevent common attack attempts.

Hope this has been informative and insightful. For more information, refer to these wonderful and insightful RFC documents below.

• RFC6819 - OAuth 2.0 Threat Model and Security Considerations
• Accessing Protected Resources
• RFC8705 - OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens
• RFC8707 - Resource Indicators for OAuth 2.0
• RFC8414 - OAuth 2.0 Authorization Server Metadata

As 10G and Multi-Gig capable hardware becomes more affordable, it made sense to do some upgrades to my home networking in order to transition to these newer and faster technologies. Many routers, switches, and Wi-Fi devices these days are supporting the IEEE 802.3bz standard. This standard allows for increased networking speeds of up to 2.5Gbps or 5Gbps over a typical 1Gbps connection. All while utilizing standard Cat5e cabling found in many residencies and offices. In this post, I'll go over some of the hardware changes since my last post, Virtualize pfSense for Google Fiber - A Dream Networking Stack, to the core networking for my home in order to support these advanced technologies. As well as some configuration changes to make the networks secure and reliable.

Hardware Details

All of my networking equipment sits in a closet on my ground floor encased in a NavePoint 9U network rack enclosure. From top to bottom, we have the following:

• Patch Panel 24 Port Cat6A with Inline Keystone
• Netgear 10-Port 10G / Multi-Gigabit (MS510TXUP)
• Lenovo ThinkCentre M920x Tiny
• Synology DS1821+ 8 Bay NAS
• CyberPower OR500LCDRM1U 500VA/300W UPS

The AC powered fans that came with the NavePoint enclosure were too loud for my liking, so I opted to purchase and replace with AC Infinity S7-P Dual 120mm fans. The speed on these fans can be adjusted using a controller.

Lenovo ThinkCentre M920x Tiny

I upgraded from a Qotom Q355G4 to a Lenovo M920x Tiny for use as my pfSense router and firewall appliance. This little 1L PC is capable of some serious computing. It was manufactured from Lenovo with a 6 core Intel i7-8700 processor and a 128GB NVMe SSD. I opted for the lowest RAM possible and swapped that with 32GB of my own. The unit can handle two NVMe solid state drives, so I installed another for more space and for some redundancy in the form of periodic backups.

The specs of this tiny PC are quite impressive and include the following:

• Intel Core i7-8700 vPro (6 core / 12 thread, 3.20GHz up to 4.60GHz with Turbo Boost)
• 32GB DDR4 2666MHz (SO-DIMM)
• SK hynix Gold P31 1TB PCIe NVMe SSD
• 128GB NVMe SSD
• SuperMicro AOC-STGN-I2S Dual Port 10GB SFP+ NIC

Upgrading to 10G Routing

I originally configured my Lenovo M920x Tiny with an Intel i340-t4 and had been using this with a similar configuration as my Qotom unit. I found that transitioning this unit to 10G was quite easy and cost effective. And many of the parts can be found on Ebay for cheap. To do this upgrade, I did the following:

• Swapped to the 01AJ940 PCI-E x16 riser card (originally only a x4 riser card was installed).
• Installed the SuperMicro AOC-STGN-I2S NIC
• Purchased a 3D printed bracket from a fellow Redditor.

The 3D printed bracket is not a required item. But the PCI bracket used has to be specific for these Lenovo Tiny devices. Not just any generic low-profile bracket will work with the case. You may remove the bracket all together. That said, it makes the network card much more stable in the case. Preventing it from moving when inserting and removing transceivers from the SFP+ ports.

My hypervisor (Proxmox) is installed on the 1TB NVMe drive. The other drive is used to store periodic backups of the running VMs.

Netgear MS510TXUP

For my root router, I converted from a Ubiquiti UniFi Switch 8-Port 150W (US-8-150W) to a Netgear 10-Port 10G / Multi-Gigabit (MS510TXUP) switch. This is a pretty interesting switch as it offers four 1G/2.5G/5G/10G Ethernet ports, another four 1G/2.5G Ethernet ports, as well as two SFP+ 1G/10G ports. This allows transitioning the network as my hardware on the network supports these newer higher speeds.

In addition, all 8 Ethernet ports have Power Over Ethernet capabilities utilizing IEEE 802.3bt (Type 3) POE++. Each port can provide a maximum power output of 60W. With the switch capable of providing up to 295W of power in total!

This Netgear switch can either be configured using a management UI served through the device. Or through a cloud interface similar to the Unifi controller. Unlike the Unifi product line, the cloud service provided by Netgear, Netgear Insight, is a paid service. But it does seem priced reasonable well for home users at $9.99 / year, or $22 / year for the Pro service. I decided to forgo the payments and manage through the device.

Network Details

Generally, a conventional router will have a WAN and one or more LAN ports. My setup is a bit different in that both WAN and LAN traffic are transmitted through the same network port. In order to achieve isolation, I use VLANs to segment the traffic through the root switch.

Here is a table matrix of the physical ports on the root switch.

The first port is used for egress and ingress traffic through my ISP, Google Fiber. Ports 2-8 are used for local traffic. Finally, we have the SFP+ ports, 9-10. Which are dedicated to routing. Currently, I only connect one 10G port to my router, which is plenty for my routing needs. In the future, I could connect another SFP+ cable and configure link aggregation for a theoretical speed of up to 20G.

I'm still in the process of upgrading the cables in my home. I hope to have multi-gig or 10G available to my lab in the near future. When WiFi 6E becomes reasonably priced, I plan on upgrading my two access points, on ports mg2 and mg3, and will have plenty of bandwidth capacity and power for those.

Here is a breakdown of the status of POE from the switch management UI. The first port, mg1, powers the Google fiber jack. The following two ports power my Unifi Access points. One great feature of this Netgear switch is that you can power cycle the POE from the UI. So if an access point is acting up, you can easily cut power to it and have it restart from your browser. In addition, you can monitor power consumption of the POE devices.

VLANs and Subnets

I use VLANs and subnets to isolate and control cross traffic for all of my networks. The benefit of this is that I can configure which devices can communicate with one another through firewall rules. As well as specify which interfaces traffic is able to propagate through.

For firewall rules, I completely isolate the guest WiFi from the rest of the other networks. In addition, I have some rules which establish trust boundaries on what devices are able connect to my management network. The management network consists of all the BMCs and OOB management devices and accesses to those remote services. This includes the management IPs for the Unifi access points, managements UIs for switches, iDRACs, and IMMs. Finally, the IOT network, which includes a secured home surveillance system, has a set of whitelisted IPs that can access the devices. These devices cannot reach out to the internet themselves.

A pfSense package I use to supplement security rules is pfBlockerNG. Which allows you to set country specific GeoIP block lists. I decided to select every country which the United States has current sanctions with and block them on all interfaces.

Google Fiber IPv6

Google Fiber will give out a /56 prefixed block of IPv6 IPs for delegation. I found this to be a little tricky to setup. The settings in pfSense can be finicky and require some playing around with. And I have experienced many instances where restarting my router has caused me to lose my IPv6 reservation. The only way I have found to get a new reservation is to power cycle the Google fiber jack itself.

That being said, when I do have an IPv6 reservation, I subdivide that /56 prefix into separate /64 prefixed networks. This allows for up to 256 different /64 prefix networks. Which is just an astonishing amount of contiguous IPs coming from IPv4!

Here is what my WAN DHCP6 setting look like.

Then for each internal network interface, set IPv6 Configuration Type to Track Interface. Under the Track IPv6 Interface section, select WAN as the interface to track. And set a unique Prefix ID for each interface. Finally go to Services > DHCPv6 Server & RA, under Router Advertisements for each interface, make sure Router Mode is set to Assisted.

One thing to be aware of when using IPv6 is to make sure your firewall rules are taking the IPv6 protocol into account. Since you no longer have NAT to block direct access to local hosts, this becomes very important.

Hypervisor Configuration

Instead of running pfSense directly on my Lenovo M920x Tiny, I opted to virtualize it using Proxmox as a hypervisor. The configuration is very similar to what I had prior in my last post. But with a few important modifications.

The first change I made is instead of virtualizing the network ports, I am now passing the PCI-E device directly through to pfSense. This allows a bit better performance as well as the ability to enable hardware checksum offloading. Since I am also using another virtual network in Proxmox to connect other VMs running along side pfSense, the virtual port on the pfSense VM to that Linux bridge must be of type Intel E1000 and not VirtIO. I have found that hardware checksum offloading does not work well with the later.

Here we can see the Linux bridge, vmbr1. This is used as a virtual network switch for other VMs on the hypervisor to communicate on. For example, my virtual machine for running Unifi Controller and Pi-Hole software, are connected through this virtual bridge.

The Lenovo M920x Tiny has an integrated Ethernet port, labelled eno1 here. This is used for management access to the hypervisor. If I ever am unable to reach pfSense or Proxmox through the root switch, I am able to connect a cable directly to this port. Then I have access to the Proxmox UI, where I have a Kali Linux VM available to start to use for troubleshooting.

Finally, instead of running LXC containers through Proxmox, I am running all services within virtual machines. I used to have privileged LXC containers running Docker engine and Pi-Hole and Unifi Controller Docker nested containers. But had issues upgrading Proxmox as it is not technically not a supported configuration through the hypervisor. I have found that VMs are a bit easier to self-contain and transfer, if need be, anyway.

Routing and Security

DHCP is configured in pfSense such that hostnames are registered to the DNS forwarder. This allows me to use DNS to access and reference all the internal hosts. For each network, I allow DHCP to assign from half of the set of IPs of the subnet mask. The other half of IPs of the subnet can be used for static IP mappings.

One pfSense package that is very nice to have is arpwatch. With this service, you can monitor your system for hosts communicating on your networks. It will detect new MAC addresses discovered through ARP messages and log the timestamp, hostname, MAC, and IP addresses of when that host was discovered. You can also have it email you if a new host is discovered.

For security on my Netgear MS510TXUP root router, I have DHCP Snooping enabled on the ports and VLANs. With this, I can make sure that only pfSense is handling DHCP requests. Any packets for other rouge DHCP services would be dropped.

The Netgear switch has a few other services which can be used to secure your networks. Such as port security and Dynamic ARP Inspection. This can be used to protect against MAC and IP address spoofing and ARP based attacks.

There are also a few services which can be enabled, such as Spanning Tree Protocol (it supports MSTP and RSTP) as well as IGMP snooping and Multicast VLAN Registration (MVR). These services can help optimize broadcast and multicast traffic forwarding through your internal networks.

Conclusion

I am becoming satisfied with the availability of new products supporting upwards of 10G. These products are becoming reasonably affordable for home users. Running pure 10G networks can be very expensive. Not only does cabling have to be upgraded, but the hardware and power required to run at 10G speeds can be prohibitive for most enthusiasts. These multi-gig (2.5G and 5G) solutions provide a nice in-between for getting past the gigabit wall we all have been experiencing for some time now.

I hope this post has been informative. I believe that it is very important to understand these networking fundamentals. Especially for professionals in the IT industry. Being able to troubleshoot networking issues and understand infrastructure is an awfully important skill to have. And that is reason alone to over engineer your own home networking :-) Happy learning!

Products

The following are the products used in this post.

When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network.

When installing any Linux distribution, an often overlooked and misunderstood component of the configuration process is setting up the storage of the system.

In the early days of Linux, there were a few options to consider. Such as whether to format your partitions as EXT3, XFS, or if very adventurous, a variant of the ReiserFS linage. If one wanted some form of software RAID, the only option to consider was Linux mdadm. And setting up system partitions, such as a boot partition, was straight forward since things like UEFI and the EFI system partition were not standard.

Today, we have a lot more options to consider. With the development of more advanced and all-encompassing filesystems such as LVM, Btrfs, and ZFS, we are able create more purpose built storage architectures for our specific use cases. As an example, the physical storage composition of our systems can vary from containing drives dedicated for capacity and some for speed. Understanding these storage technologies can be important for increasing the performance of our systems as a whole. In addition, we can use these filesystems to protect ourselves from losing data due to drive loss, a failed system update, or even user error.

In this post, we are going to look at the current state of storage within Linux. We will evaluate LVM, Btrfs, and ZFS from the perspective of a desktop user. And we will look at the pros and cons of the different technologies and how we can use them to our advantage.

Logical Volume Manager (LVM)

LVM offers an additional management layer in between the physical drives and the filesystems initialized on the system.

There are a few key concepts within LVM to first understand: physical volumes, volume groups, and logical volumes. A physical volume represents any storage device that is initialized within the context of the LVM subsystem. Volume groups are made up of physical volumes, and represent the total combined capacity of a particular LVM subsystem. Finally, logical volumes utilize the storage pools defined by the volume groups. Logical volumes can be initialized with a particular RAID, mirroring, or striping configuration for data redundancy and performance.

By default, logical volumes will reserve from the volume group all of the space specified during creation. We can use thin pools, which are special types of logical volumes, to dynamically grow and reserve space as the system needs. This allows us to over allocate space within a Linux storage system.

Advantages of LVM

LVM can provide significant advantages compared to using only a traditional filesystem.

One of the first advantages is that LVM provides a way to dynamically add or remove storage from a system. To do this, we extend the volume group to add a new physical volume. Afterwards, the logical volume can be resized, which is analogous to resizing of a partition. And finally, the filesystem can be resized to gain the extra storage at that mount point. This can be a very convenient method of expanding storage, for example, on a virtual machine that has run out of space.

Another important feature is that of snapshots. A snapshot can be created on a live system. And it can be referenced at a later point and be written to or reverted to. Snapshots are a great way to prevent errors if created before upgrading system packages, or even to protect from deleting an important file.

Many systems these days will have both SSDs for faster storage and traditional spinning HDDs for capacity. With LVM, we can leverage this by setting up caching on logical volumes such that read and write performance is increased while also allowing for full capacity of the data drives. It is important to note that some configurations of a writeback cache will require that the underlying storage is reliable; such as those in a redundant RAID configurations and on backup power. Because a loss of such a writeback cache drive could cause filesystem data loss.

LVM has been apart of the Linux ecosystem for a long time. Due to its maturity, LVM is found in many popular distributions by default. As a product of Red Hat, LVM plays a large role in that of CentOS as well as Red Hat Enterprise Linux (RHEL). LVM can be used in combination with other technologies, such as Red Hat's VDO, to offer things like compression and deduplication. Finally, with LVM, you still may use any of the traditional filesystems such as XFS and EXT4, which are familiar by the majority of users in the Linux ecosystem.

Disadvantages of LVM

One disadvantage of LVM, which one should be aware of, is the issue of bit rot. Bit rot is due to data degradation at the physical drive level. LVM RAID utilizes the traditional mdadm software RAID which comes with caveats for finding and repairing such data degradation. According to the lvmraid manpage:

The repair mode can make the RAID LV data consistent, but it does not know which data is correct. The result may be consistent but incorrect data. When two different blocks of data must be made consistent, it chooses the block from the device that would be used during RAID intialization. However, if the PV holding corrupt data is known, lvchange --rebuild can be used in place of scrubbing to reconstruct the data on the bad device.

To summarize, with the LVM stack and a RAID configuration, we are able to detect inconsistencies in a replicated block of data. Yet the LVM utilities are only able to make a best guess as to which of the blocks is correct to make the system consistent. If the user knows which of the blocks is good, they may manually rebuild the block. We will see in the next sections how Btrfs and ZFS solve this issue using additional filesystem metadata.

The final disadvantages are due to the way snapshots work in LVM. Like logical volumes, a snapshot is exposed as a block device. Transferring a LVM snapshot for backup purposes, in an incremental and efficient way, can be difficult because of this fact. Also, there are several reports of having significant snapshots on a system causing performance degradation. Though I have not found any reputable investigations of this issue, so your results may vary. Maybe this is something we can look into in a future post ;-)

Summary of LVM

As we have seen, LVM is a great tool for expanding the functionalities of traditional filesystems. It represents an old-school modular way of configuring a storage stack within Linux. Where features are added by creating additional logical abstractions of the storage in-between the physical disk and the filesystems.

LVM is a great option for use cases where bit rot is not a high priority issue. Or if the underlying storage is highly reliable, as the case may be for public cloud environments, such as AWS or Google Cloud.

Btrfs

Btrfs is a newer filesystem developed specifically for the Linux ecosystem. It offers many of the same benefits of LVM, but the features are included in the filesystem itself.

A volume is defined during initialization of the filesystem using mkfs.btrfs. We pass in any number of block devices, as well as define the data replication strategies of this volume. Different strategies can be defined for the filesystem metadata and the actual file data of a particular Btrfs volume.

The way Btrfs handles mirrored RAID profiles is different than is traditionally defined. Btrfs works on the concept of data copies for mirrored RAID profiles. For example, if a RAID1 volume is created with three equally sized disks, each data block will have one associated copy, and the volume will be able to handle a single disk failure. This is different than a traditional RAID1 configuration in which all three disks would hold the same copy of the data, and the system would be able to handle two disk failures.

After initialization of the volume, a default top-level subvolume is created. A subvolume is considered a namespaced portion of the volume and can have it's own snapshots or subvolumes anywhere within its own mounted directory structure. As a general rule, a subvolume is created to isolate the data within it, and can have separate mount options and snapshots.

A snapshot is a special type of subvolume. A snapshot is created by targeting an existing subvolume. Afterwards, a Copy-on-write pointer to the subvolume is created as it exists at that moment in time. This uses no extra storage and is a very quick operation. The two subvolumes will appear to contain the same data, but the files within can be updated or deleted without impacting the other. We can then revert to a particular snapshot by mounting it's subvolume and removing the original.

Advantages of Btrfs

Unlike LVM, Btrfs generates checksums for each data and metadata block of the filesystem. With this checksum, Btrfs is able to detect silent corruption from bit rot, and fix it if a valid block has been replicated to another storage device. This is done automatically on all reads of data as a system is being used. And should be run periodically with the scrub command, to verify data that is rarely read from the filesystem.

Btrfs uses a replication strategy in which chunks are balanced across any number and size of devices in a volume. With traditional RAID, it's recommended to keep device sizes the same. With Btrfs, as long as the amount of devices with free space satisfies the data replication strategy of the volume, Btrfs will handle balancing the data across all of the storage devices. Since Btrfs is flexible in this way, converting to a different RAID level, and adding or removing devices, can be done very easily by using the btrfs balance command.

A strong advantage of Btrfs is its inclusion of mount options which are available for each subvolume. Btrfs offers mount options which can enable ssd optimizations, discard support, as well as options for disabling checksums for an entire mount point. Another popular option is that of compression. Zlib compress is enabled by default, and the filesystem will use heuristics to determine whether a file should be compressed or not and mark it as such. Similar to converting RAID levels, we can use btrfs balance to re-write our data using a different compression algorithm.

The final advantage Btrfs brings to the table is the ability for conversion of Ext3, Ext4, and ReiserFS filesystems to Btrfs. Running the btrfs-convert utility will create a Btrfs subvolume with the existing filesystem data on it and a default subvolume will be snapshotted from it. This operation can be completely reverted back to the original filesystem, if needed.

Disadvantages of Btrfs

Btrfs has a few notable gotchas which a user will want to know about.

The first disadvantage is the stability of the parity RAID5 and RAID6 implementations. Utilizing these causes a system to be vulnerable to drive and power failures. In which the filesystem may become corrupted beyond repair. This is known as the "write hole".

Parity may be inconsistent after a crash (the "write hole"). The problem born when after "an unclean shutdown" a disk failure happens. But these are two distinct failures. These together break the BTRFS raid5 redundancy. If you run a scrub process after "an unclean shutdown" (with no disk failure in between) those data which match their checksum can still be read out while the mismatched data are lost forever.

According to the Btrfs wiki quoted above, a "write hole" event may happen with RAID5/6 in which an improper shutdown could cause a mismatch between the parity and data to happen. The general advice from developers, if an individual wants to utilize a parity RAID implementation, is to utilize a RAID1 type profile, for example RAID1C3, for the metadata. And then RAID5 or RAID6 can be used for the file data. This way, in case of a write hole event, the filesystem metadata is protected while the actual filesystem contents may be scrubbed immediately on boot in order to make the filesystem consistent again.

Another gotcha concerning RAID that could catch a user off-guard, is the fact that a RAID1 volume may only be mounted once as read-writable in a degraded state. For example, if a two disk system experiences a single drive failure, the system may only be mounted as degraded once until the drive is replaced. Obviously, it is always advised to replace your as soon as possible. But this is an important limitation to know if you are expecting your systems to be functional before redundancy is fixed.

Finally, Btrfs does not currently support any type of caching to faster storage. A user is free to mix storage types in a single volume, but the filesystem will not do any type of prioritization of the data onto the faster drives.

Summary of Btrfs

As we have seen, Btrfs is a very exciting filesystem which provides many advanced functionalities to the Linux ecosystem. It combines a lot of wanted properties that exists in storage manages, like LVM, into a flexible filesystem solution.

Btrfs offers a convenient method of creating on-demand snapshots of a system. It also fixes the issue of bitrot, by establishing checksums, as a means of determining whether a block has been corrupted. And it uses redundancy profiles in order to fix any corrupted data. Btrfs has flexibility of how it can represent data on disk and offers convenient methods of converting from one redundancy profile to another. As well as converting from existing filesystems, such as EXT4 or ReiserFS.

As Btrfs is developed, it will continue to grow and improve on the stability of its features. Already, Fedora has changed the default filesystem of it's distribution to Btrfs in Fedora 33. So this is an exciting storage technology which will only increase in popularity in the desktop space in the upcoming years.

ZFS

ZFS is an open-source storage technology that came out of Solaris in the early 2000s. In 2005, under new control of Oracle, it was transferred to a closed-source model. At this time, various forks and ports of the ZFS specification came to fruition. The development of these ZFS implementations are now under the umbrella of OpenZFS.

ZFS is a complex filesystem with many different concepts which are important to understand before setup and deployment.

The first concept is that of vdevs. In ZFS, vdevs encompass a set of disks and can have a particular redundancy profile associated. A vdev supports mirroring, single parity (RAIDZ1), double parity (RAIDZ2), and triple parity (RAIDZ3) block redundancies.

Another important concept is that of a zpool. A zpool has one or more vdevs underneath it to provide the underlying storage. A zpool does not have any particular redundancy associated; that is the responsibility of only the vdevs. A zpool will distribute writes across the vdevs, mostly according to the free space available to each at the time. But it is important to understand that there is no guarantee of how the data will be written and should not be confused as true data stripping. A zpool is flexible in that vdevs under it can be dynamically added to the pool and can be of varying sizes and use any redundancy technique.

A third concept is that of datasets. A dataset is very similar to the concept of a subvolume within Btrfs. It can be used to subdivide the data within the file hierarchy so that a snapshot of the dataset will only contain what is within that dataset. A dataset can also be used to define different filesystem properties, such as compression, encryption, and even deduplication. As an example, here is one the datasets defined for one my desktop:

[patrick@summit ~]$ zfs list -o name,mountpoint
NAME                                   MOUNTPOINT
bpool                                  /boot
bpool/sys                              /boot  
bpool/sys/BOOT                         none
bpool/sys/BOOT/default                 legacy
rpool                                  /
rpool/sys                              /
rpool/sys/DATA                         none
rpool/sys/DATA/default                 /
rpool/sys/DATA/default/home            /home
rpool/sys/DATA/default/root            /root
rpool/sys/DATA/default/srv             /srv
rpool/sys/DATA/default/usr             /usr
rpool/sys/DATA/default/usr/local       /usr/local
rpool/sys/DATA/default/var             /var
rpool/sys/DATA/default/var/lib         /var/lib
rpool/sys/DATA/default/var/lib/docker  /var/lib/docker
rpool/sys/DATA/default/var/log         /var/log
rpool/sys/DATA/default/var/spool       /var/spool
rpool/sys/DATA/default/var/tmp         /var/tmp
rpool/sys/ROOT                         none   
rpool/sys/ROOT/default                 /

The final concept is that of zvols. A zvol exists at the same level as a dataset and is used to expose a raw block device to the system. This can be helpful if you want to have, for example, a separate swap partition on top of ZFS.

Advantages of ZFS

ZFS is quite an advanced filesystem and has support for more features than just about any filesystem to date. Like Btrfs, ZFS is based on the concept of Copy-on-write. The greatest benefit of this is that it allows for instantaneous live snapshots to be created from any dataset. Unlike Btrfs, ZFS takes the concept a step further and applies CoW to each data write operation.

Copy-on-write in ZFS isn't only at the filesystem level, it's also at the disk management level. This means that the RAID hole—a condition in which a stripe is only partially written before the system crashes, making the array inconsistent and corrupt after a restart—doesn't affect ZFS. Stripe writes are atomic, the vdev is always consistent, and Bob's your uncle.

According to an article by Ars Technica, entitled ZFS 101—Understanding ZFS storage and performance, due to the way that ZFS commits data to disk using Copy-on-write mechanisms, the filesystem is not affected by the write hole problem unlike Btrfs. In fact, the parity implementations in ZFS, denoted as RAIDZ, are highly lauded by the community and are considered very stable.

An additional advantage ZFS brings to a system is advanced caching hierarchy for both read and write operations. Traditionally, filesystems let the kernel page cache handle caching of recently used blocks and file metadata. ZFS takes another route and utilizes in-memory caches using more advanced algorithms. This read cache is considered the Adaptive Replacement Cache (ARC) and is the main reason that ZFS is known to like a lot of RAM. When an eviction of the cache occurs, one can configure a high performance L2ARC device, which extends this read cache to hold more frequently read data.

For speeding up synchronous write operations, ZFS utilizes a special device known as the Secondary Log (SLOG) device. A user may register a high endurance and write optimized persistent storage device to handle the ZFS Intent Log (ZIL) on the SLOG device. This ZIL is simply a journal of write transactions to the ZFS pool. The ZIL may be referenced on crash of a system, and is used to keep the filesystem consistent and the data within it reliable. It is important to note this cache is only for synchronous write operations, which are used heavily within databases and virtual machine virtual disks.

Finally, ZFS offers many different features which may optionally be enabled on a per dataset definition. Compression, deduplication, and encryption can all be enabled on each dataset. Likewise to Btrfs, ZFS checksums all filesystem data in order to protect against bit rot.

Disadvantages of ZFS

ZFS is a without a doubt a tremendous filesystem with many features and a stable implementation. That being said, there are a few disadvantages which a user should be aware of.

The first is that ZFS is under the open-source CDDL license. Unfortunately the CDDL license is considered incompatible with the GPL license. This poses problems because ZFS is not legally allowed to be included with the Linux kernel. Instead, it's advised that a user should use the ZFS Dynamic Kernel Module (DKMS) packages to load the required kernel code on boot. This present the following problems:

• The ZFS kernel modules must be included in the initramfs.
• The ZFS kernel modules must be independently rebuilt for each new kernel release.

The first issue only presents itself as a problem if the user compiles their own kernels. In which case, they need to always remember to include the ZFS DKMS packages with their custom kernels, especially if booting from or have root mounted as ZFS. The second is a problem that many users of rolling distributions, for example Arch Linux, can face. If a new Linux kernel is released, there may be a period of time when a system cannot be updated due to conflicts. This is due to the independently maintained ZFS DKMS packages lagging behind. It's an unfortunate reality due to a simple license incompatibility.

The final disadvantage has to do with expanding the storage of a zpool for RAIDZ configurations. ZFS is not as flexible on adding and removing storage as Btrfs or LVM. As detailed in the article entitled "The 'Hidden' Cost of Using ZFS for Your Home NAS", a vdev cannot be easily changed after initialization of a parity RAID configuration. This means that a disk cannot be added later down the road when you are ready to expand storage. One cannot simply add a disk and rebalance like Btrfs. You will need to create an additional vdev, ideally of the same RAIDZ profile type, and add that vdev to the zpool. One other option, is replacing each drive, one at a time, with a larger drive, and expanding a vdev that way. But this is wholly inefficient in time as a re-silvering is required for each drive replace.

Summary of ZFS

ZFS is an amazing swiss-army knife for storage. It is able to handle all storage concerns well and offers many features that one would want in a filesystem. Its feature sets were the basis of comparison to newer filesystems like Btrfs. And will continue to be used and have a strong following by storage enthusiasts.

ZFS makes a great use case for NAS devices where data integrity is critical. And excels with large storage pools with caching devices used for boosting read and write performance. It is also used in high regard as local persistent storage for a cloud platforms such as used by Proxmox. Where it is able to handle on-demand snapshots and offer high performance storage configurations.

Conclusion

It's certainly an exciting time to be around with such accessible storage technologies to the end user. With projects like LVM, Btrfs, and ZFS in use and development, it is important to understand these technologies to see what their use cases are in relation to the others. And more importantly, to learn from them the concepts that they bring to the Linux ecosystem. And how we can set up our own systems for better reliability, performance, and usability.

Thanks for reading. I really appreciate feedback on what you think about these conclusions, and what your own use cases, both at work and in your own home lab, have been!

Our goal in this post is to setup a secure home VPN server which we can use to connect our phone or laptops remotely. This will allow us access to all of our home networked resources such as printers and security cameras. We will look at how to configure a pfSense router for remote access using OpenVPN. These instructions will target residents who have a dynamic IP address which may change without notice.

In order to properly configure a VPN server, we will need to establish a DNS record which will track our IP address. Luckily, there are free services we can use to register with our pfSense router in order to do exactly this.

Dynamic DNS

The Dynamic DNS (DDNS) pfSense service allows us to create a configuration such that when our IP address changes, our DNS entry will be updated accordingly.

There are many DDNS services which allow you to register a free subdomain to track your IP address. For my setup, I use freemyip.com which allows registration of a DNS entry, and a convenient method for updating it when it changes.

After you register your unique subdomain, you will be given an update URL with which to call when your IP address changes. We can do this automatically within pfSense.

Go to the Dynamic DNS service within pfSense and click Add to start the process of creating a new DDNS client.

The setup is very easy. Simply paste the update URL into the form, and make sure that the Verify SSL/TLS Certificate Trust is selected as well.

OpenVPN Server

After establishing our DDNS client, we can now begin setting up the remote VPN Server configuration.

The easiest method of doing so is going through the OpenVPN Remote Access Server Wizard. The official Netgate documentation has a very comprehensive example which goes through all the options for the wizard. It is recommended to follow this in order to create a Certificate Authority, a corresponding Server Certificate, and to configure the OpenVPN Server itself.

For my setup and the rest of this guide, I am using local users as the authentication type instead of any remote authentication available.

Creating OpenVPN Clients

Once an OpenVPN server configuration is created, we can now create local users which will represent our clients to connect remotely.

It is recommended to have one user per device. For example, if you have a laptop and phone which you would like to connect, you should create a user for each and not share. This allows us to easily revoke the users client certificate in case the device is lost or compromised.

You can create a new user by going to System -> User Manager.

You can go ahead and create a new OpenVPN client certificate by clicking the checkbox and selecting your Certificate authority you referenced in the OpenVPN setup wizard above.

Export OpenVPN Configurations

Once we have our users created, we can export any OpenVPN configuration using the OpenVPN Client Export Package available through the Package Manager in pfSense.

Once the package is installed, the export can be accessed by clicking the Client Export tab within the OpenVPN service page.

Make sure to input your DNS entry in the Host Name input, after selecting Other as the Host Name Resolution option. You will need to do this each time you are exporting a new configuration file.

Scrolling below, you may select a configuration file type to export depending on your needs.

If you use NetworkManager, the Networkmanager-openvpn plugin works rather well to import a configuration file directly. There exists apps for Android and iOS as well.

I recently moved to my first home and immediately started planning improvements to the networking situation. This is a story of planning and executing on a networking stack re-design utilizing Google Fiber, pfSense virtualized in a Proxmox hypervisor, and some Ubiquiti products to boot.

Fixing the Wiring Cruft

Upon moving in, I wanted ethernet runs to the rooms used for my office and lab. Fortunately, the home was wired with Cat5e homeruns to the rooms but they were terminated as RJ11 (phone) jacks. So first things first, I re-terminated the Cat5e as RJ45.

There was no way I could have relied on WIFI for all my hardware and service needs. So this was an easy first win, in my books!

Google Fiber

I have access to the amazing Google Fiber in my area. So this would be the foundation for me to build my networking on. I used the Google Fiber router for a few months but always felt the pull to go back to the more advanced routing of pfSense which I had deployed in the past. It is difficult to not feel like you are missing out with proper VLAN tagging and management, firewall rule definitions, VPN integration, and IDS which pfSense offers in its ecosystem. So with that in mind, I began my R&D.

Hardware Utilized

pfSense runs nicely on commodity hardware (although some would argue as long as Intel NICs are used, along with a CPU which supports AES-NI). In the past, I used a Qotom Q355G4 to run pfSense and it worked reasonably well. Outfitted with 8GB of DDR3, 4 x Intel I211 NICs, and an efficient Intel i5-5200U processor, it is able to handle routing traffic along with a surprising amount of layer-2+ services in parallel.

In addition, I purchased a US-8-150w switch and a UAP-IW-HD access point from Ubiquiti to supplement the AP-AC-PRO I had already.

Virtualizing pfSense

The idea of running pfSense under Proxmox was very appealing to me. The benefit being able to manage pfSense in a VM and to be able to offload some of the network interface details to the hypervisor. This would allow pfSense to only concern itself with the routing and application layers. Also, I will be able to easily and, most importantly, quickly backup and restore the VM during upgrades or testing of functionality. Finally, this would allow me to deploy additional services such as Pi-Hole and the Unifi Controller on the router itself.

This is the design I came up with:

There are some interesting concepts in this design which make it really resilient and performant. The first being the usage of a static management port connecting to Proxmox. This allows an admin to connect directly to the host, using either SSH or the Proxmox Web GUI, for debugging of any VM or hypervisor detail. So if something goes wrong on the network, I can always patch in directly to the router and fix it.

Next, the usage of Open vSwitch bridges and bonds makes it really simple to create aggregation groups and virtualized switches. This hides a lot of the implementation details from pfSense and thus configuration is much simpler. In this case, I will be using 1 port for WAN which goes to the Google Fiber jack, 2 ports for LACP to the US-8-150W switch for LAN, and finally the last port is reserved for the management access.

The final benefit of this design is the usage of LACP to the switch. This not only increases the theoretical throughput to the router to 2Gbps but also adds some failover in case one of the links in the group goes down.

One proposal I had planned was to use an OVS IntPort in order to tag all outbound packets with VLAN2, as required by Google Fiber. Unfortunately, this did not work as planned and I eventually did this configuration in pfSense, as we will see below.

Executing on the Proposal

I setup the router separately before switching out the Google Fiber network box. Obviously, you will want to create the pfSense VM before exchanging the hardware. Next, I configured the Ubiquiti gear and the LACP trunk ports to the switch. Last, I installed all supplemental services, such as Pi-Hole.

Promox Installation and Configuration

Installation of Proxmox is very straightforward. Here is my final network configuration as indicated in the proposal.

You should put the OVS bond interface in active-backup mode until you can configure the downstream switch for LACP. Then and only then, change this setting to LACP (balance-tcp). This is where the management port comes in handy!

You may also notice that the NIC ports on the Qotom are not represented on the hypervisor host in the order that they are laid out on the chassis. I put a comment on each interface to know which physical port it is mapped to.

pfSense VM Creation

I followed this guide for virtualizing pfSense within Proxmox. The following are the resulting Virtual Machine details I used for pfSense.

In the advanced options, when setting up the CPU, you may enable the AES flag for the VM, if your processor supports it. Also, I decided to use ballooning for the memory configuration from 1GB minimum to 4GB maximum. Finally, make sure you enable VirtIO on the HDD and network devices as this will greatly affect performance. As in the guide above mentions, I did have to disable hardware checksum offloading in pfSense.

Also, be sure to enable the option to start the VM on boot and set a low order priority so that it starts before the other VMs and containers you have running on the hypervisor.

Replacing the Google Fiber Box

Replacing the Google Fiber network box was straightforward but did require resolving a few issues.

Like I said prior, my initial goal was to use the Open vSwitch IntPort to tag the outbound packets with VLAN 2. But this turned out to not work for a reason which I did not bother to spend time on. Instead, I followed this guide which configures the tagging within pfSense and the IntPort is mostly default and unused. As a side note, you could probably get away with using a simple bridge for the WAN in Proxmox instead.

Within pfSense, create a VLAN interface with tag 2 and priority 3. Set the parent interface to the current WAN network interface.

In the Interface Assignments page, change the WAN assignment to the new VLAN 2 tagged interface.

Finally, you will want to power down the Google Fiber fiber jack for several minutes. This seemed to be required for pfSense to receive another WAN IP address from Google upstream.

Unifi Switch Configuration

At this point in time, I had internet access but I still had the bond configured as active-backup. So, I created a LXC container within Proxmox and a nested Docker container within it to run this Unifi Controller container.

After setting up the controller and adopting all of the Ubiquiti gear, I setup LACP on the Unifi switch.

Then I used my management port to change the bond mode to LACP (balance-tcp) within Proxmox. And viola!

The Unifi Controller service is really powerful and something that I had not used prior to this endeavor. It has a lot of insight into the network details and has a great overall user experience. I will definitely be spending a lot of time in this UI in the future.

Finalizing the Configuration

In addition to all this, I created another LXC container on the Qotom to run Pi-Hole. Then set the pfSense default DNS server to be the Pi-Hole and set the DNS resolver to forward all requests to it.

The final static IP mappings and network services which are accessible from the LAN look like:

• pfSense Web UI - https://192.168.1.1
• Pi-Hole Web UI - http://192.168.1.2/admin/
• Proxmox Web UI - https://192.168.1.3:8006
• Unifi Controller Web UI - https://192.168.1.4:8443/

Conclusions and Future Work

At this point, I have so much more insight and configurability into my home network. The Ubiquiti products make it really simple to see weak points in a home WIFI network such as those caused by interference or inadequate coverage. In additional, pfSense has loads of functionality which I still need to go through.

For those that are wondering, I am seeing 940Mbps download and 750Mbps upload and the Qotom is not breaking a sweat.

Finally, there are certainly improvements I will make in the future, such as:

• Intrusion detection using Snort or Suricata
• Forwarding firewall events to Elasticsearch
• Proper VLAN tagging
• OpenVPN
• Guest WIFI

In this post, we will look at common security standards for email sending in order improve email delivery placement. We will look at methods, such as DKIM and Sender Policy Framework (SPF), which we can use to establish trust between a sender and receiver. We will also look at a few tools which we can use to test and verify that a domain configuration is working properly.

Why is Email Security Important?

The Simple Mail Transfer Protocol (SMTP), which is the foundation for sending email today, was built during a time when encryption and authorization was not widely used. As more individuals, businesses, and entities relied on internet communication through email, it became apparent that protection from spoofing, spamming, and other fraudulent acts was necessary to add onto the protocol.

This brings us to our first point, email is not secure by default. It is trivial to create an email which looks like it is coming from another party. Just like it is possible to type another individuals name on a letterhead and send it through conventional mail, the same is possible with email. As a result, a set of standards for establishing trust between a sender and receiver were created.

For the above reasons, it is clear that Email Service Providers, such as Gmail and Outlook, will rank secure and trusted email higher than insecure counterparts. If you do not ensure proper security of your email and domain configuration, your email will be vastly more likely to end up in spam folders, caught by spam filters, or worse cause a domain blacklist. All of these scenarios will cause your delivery performance to be poor resulting in less opens and clicks and less outreach to your audience.

Email Security Used Today

From an email service provider and user prospective, there are several security goals that need to be addressed:

• Validate that the content of an email has not been changed from sender to receiver.
• Validate that the sender is authorized to send an email as a particular identity.

Both of these concerns are addressed using public Domain Name System (DNS) records.

DKIM

DomainKeys Identified Mail (DKIM) is used to detect email content modifications. DKIM does this by adding a domain signature to each email sent. This signature is computed using public key encryption of a hash of the headers and body of an email.

The idea is that the sender uses a private key to sign all outbound email and attaches the DKIM-Signature as a header to the email. The receiving email provider then uses this signature and a computation of the of the headers and body to verify the authenticity of the email content has not changed.

Setting up DKIM keys and DNS records will vary from system to system. Here are some example documents to get you started:

• G Suite: Enhance security for outgoing email (DKIM)
• Postfix: How To Install and Configure DKIM with Postfix on Debian Wheezy

SPF

While DKIM validates the content of an email, the Sender Policy Framework (SPF) is used to validate that a sender is authorized to send email as a particular domain or identity.

From the Common Mistakes of SPF, "the purpose of SPF is to advertise your domain's mail servers" and this done by specifying each server which is authorized to send email. This is done by using TXT records attached to the sending domain.

"v=spf1 ip4:192.0.2.0/24 ip4:198.51.100.123 a -all"

This record defines the framework version, an IPV4 address range, and an IPV4 address, along with allowing the A record to send email. The final -all specifies to drop all email sent not matching these patterns.

The SPF syntax can be found at: http://www.open-spf.org/SPF_Record_Syntax/

DMARC

Finally, there is the Domain-based Message Authentication, Reporting and Conformance (DMARC) protocol. Building on top of the last two security measures, DMARC specifies that the domains used in the SPF and DKIM validations must align with the FROM header on an email message.

In addition, DMARC allows to define what to do when a breakage in validation of SPF or DKIM occurs. For example, you can tell a sender to send a report back to you on failures of validation.

For instructions on setting up DMARC, see:

• G Suite: Enhance security for forged spam (DMARC)

Validation Tools

To make sure that your domain configurations are functioning correctly, I recommend using the following tools:

• SparkPost/Port25 - Authentication Checker
• dmarcian - DMARC Testing & Reporting Tools
• MXToolBox - SPF Record Check & SPF Lookup

In this post, we will develop a way of maintaining high availability with a two or even node count Proxmox cluster deployment. To do this, we will be using what is considered a QDevice, in the Corosync nomenclature, in order to break ties and keep a quorate in the cluster. This will allow your hypervisors the ability to operate under failure conditions which would otherwise cause outages. The use of QDevice is only recommended for non-production deployments where another full Proxmox node is not feasible. For our purposes, we will be using a Raspberry Pi for the QDevice. Let's begin!

Understanding the Benefits

Proxmox uses the Corosync cluster engine behind the scenes. The Proxmox background services rely on Corosync in order to communicate configuration changes between the nodes in the cluster.

In order to keep synchronization between the nodes, a Proxmox requirement is that at least three nodes must be added to the cluster. This may not be feasible in testing and homelab setups. In two node setups, what happens is that both nodes must always be operational in order for any change, such as starting, stopping, or creating VMs and containers, to be done.

In order to fix this, we can use an external QDevice whose sole purpose is to settle votes during times of node outage. This QDevice will not be a visible component of the Proxmox cluster and cannot run any virtual machine or container on them.

The benefits for adding a QDevice to a cluster include:

• Allowing modifications to the running hypervisor during single node downtime in a two node deployment.
• Settling disputes in even node count deployments.

Creating the QDevice

For my purpose, I am using a Raspberry PI 2B and the Arch ARM distribution for a slim Corosync QDevice node. My Proxmox two node cluster is running the latest Proxmox 6 which utilizes Corosync 3.

Installing Dependencies

On the RBP host, we will want to install the following corosync-qdevice dependency in the AUR:

• https://aur.archlinux.org/packages/corosync-qdevice/

It should be noted that at time of writing, the following dependencies of this package do not include the armv7h arch in the PKGBUILD.

• https://aur.archlinux.org/packages/libcgroup/
• https://aur.archlinux.org/packages/kronosnet/
• https://aur.archlinux.org/packages/corosync/

You can safely add armv7h to the PKGBUILD manually and expect a successful build and install from these packages.

Once the package dependencies are met, we can start and enable the corosync-qnetd systemd service:

sudo systemctl start corosync-qnetd.service
sudo systemctl enable corosync-qnetd.service

Adding the QDevice to the Cluster

On each of the Proxmox nodes, you will need to do the following.

1. Make sure the QDevice can be reached via SSH.
2. Install Corosync QDevice and QNETd dependencies.
   • apt install corosync-qnetd corosync-qdevice

Once this is completed, we can run the Proxmox Qdevice setup as below:

pvecm qdevice setup <rbp-ip>

Verification of Success

To verify, you can use the pvecm status command and see that the Qdevice has been added and that it contains a single vote as member of the cluster.

root@pve:~# pvecm status
Cluster information
-------------------
Name:             JenCluster
Config Version:   5
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Jan  3 22:13:24 2020
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.2c
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 192.168.1.131 (local)
0x00000002          1         NR 192.168.1.116
0x00000000          1            Qdevice

One important information which should be noted, is that you should remove the QDevice, as noted in the Corosync External Vote Support, before the addition of the Proxmox node to keep an odd node count in the cluster.

Smartmontools is a very well known toolset for monitoring and querying storage health. The information that the project utilizes is a part of the Self-Monitoring, Analysis and Reporting Technology System (or S.M.A.R.T.) which is a standard implemented by many modern hard drives. In this post, we are going to take a look at the code of Smartmontools and see what we can learn from it. We will also take a look at the Linux storage system hierarchy and how communication takes place from a softwares perspective. Let's begin!

Checking out the Codebase

We will be checking out the code through the official SVN repository, although a Github mirror exists as well. Revision 4934 will be used throughout this review.

Checking out the codebase is as easy as:

svn co https://svn.code.sf.net/p/smartmontools/code/trunk/smartmontools smartmontools

Building the Code

We will be using Linux for building and reviewing the code. Smartmontools uses Automake as the build system. From a clean repository, we can run the build to produce the binaries with the following commands:

./autogen.sh
./configure
make
sudo make install

Reading the INSTALL file will give details on specifics of default settings and overrides which are possible during compilation.

Reviewing the Interfaces and Data Structures

The code is written in C++ with use of a few base classes and mixins for code abstractions across the supported platforms.

The smart device and smart interface classes are the basic building blocks of the code. In addition, generic ATA, SCSI, and NVMe type classes are included for extension. These base classes are defined in dev_interface.h.

The smart_device class can be used to downcast each device object into specific implementation of the extended smart_device classes such as:

• smart_device.to_ata() returns ata_device
• smart_device.to_scsi() returns scsi_device
• smart_device.to_nvme() returns nvme_device

There are ten platforms that are supported by Smartmontools. Each of these platforms have their own unique structures and driver interfaces for querying the underlying storage. Smartmontools uses object inheritance in order to differentiate the implementations of the smart device classes and smart interfaces for each platform. These are defined in the os_<platform> cpp and header files.

In addition, the code uses a global registry singleton object to define the smart interface to use for the platform being supported at compile time. At the end of each of the platform specific source files, smart_interface::init will be implemented which will set the global interface to the platform specific implementation.

For example, in os_linux.cpp, the following code is used to register the linux smart interface as the source of truth:

void smart_interface::init()
{
  static os_linux::linux_smart_interface the_interface;
  smart_interface::set(&the_interface);
}

In conclusion, we have found how the codebase makes use of object inheritance in order to differentiate the methods of pulling S.M.A.R.T. data from the underlying storage. This provides a clean interface for extension and addition of future devices based on the platform running the executable.

Getting to the Meat of the Pie

As we go deeper into the code, we find some more interesting things to learn. For instance, the logic used to scan for devices and detect a device type. Another example we will examine is how Smartmontools is able to gather information about storage behind RAID controllers and USB devices.

Understanding Storage Device Types

One aspect we glossed over is the supported device types. There are three standards for communication with mass media used from a software perspective: ATA, SCSI, and the newer NVMe. Each offers different command structures for sending requests and receiving responses from the storage.

The ATA standards are commonly found on desktop computers. First appearing as Parallel ATA (PATA) which used the IDE "ribbon" cables. The more modern SATA devices, connectors, and host controllers continue to utilize the ATA command set. There is also the improved AHCI standard for SATA which includes additional instructions such as Native Command Queueing and TRIM support for SSDs.

SCSI is an older standard written for more than just hard disks as compared with ATA. Serial Attached SCSI (SAS) is a replacement of Parallel SCSI (SPI) and is often found in servers and enterprise storage hardware. One thing to note is that SATA drives are compatible with SAS controllers but SAS drives may not be used with SATA drives.

Finally, there is the NVM Express (NVMe) standard which is used with modern SSD devices. This standard was written from the ground up with the low latency and high parallelism of SSDs in mind. NVMe is gaining more traction in desktop and enterprise environments and will continue to gain market share as SSD devices become more economical and powerful.

Scanning Storage Devices

With smartctl, one can scan for all devices connected to the system. As an example:

$ smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device

We can see here, that two SCSI devices are detected on the system. Let's take a look at how Smartmontools detects and reports on these devices.

Within the Linux subsystem, storage devices are exposed through specific device nodes on the udev or devtmpfs mounted filesystem at /dev/. The codebase makes use of naming conventions used by Linux in order to scan for the storage devices attached to the system.

These device nodes can be deceiving and may not accurately depict whether a device is ATA, SCSI, or NVMe. For example, let's look at the device located at /dev/sdb:

$ udevadm info -a -n /dev/sdb | grep -E 'looking|DRIVER'
  looking at device '/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb':
    DRIVER==""
  looking at parent device '/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0':
    DRIVERS=="sd"
  looking at parent device '/devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0':
    DRIVERS==""
  looking at parent device '/devices/pci0000:00/0000:00:1f.2/ata2/host1':
    DRIVERS==""
  looking at parent device '/devices/pci0000:00/0000:00:1f.2/ata2':
    DRIVERS==""
  looking at parent device '/devices/pci0000:00/0000:00:1f.2':
    DRIVERS=="ahci"
  looking at parent device '/devices/pci0000:00':
    DRIVERS==""

As we can see, the /dev/sdb block device uses the sd kernel driver which handles SCSI devices. In the parent device hierarchy for /dev/sdb, we can see that the ahci kernel driver is used for the ata2 host adapter. AHCI is a standard used with SATA, and thus this is an indicator that the underlying device may be SATA.

$ lsscsi
[0:0:0:0]    disk    ATA      Samsung SSD 850  2B6Q  /dev/sda
[1:0:0:0]    disk    ATA      HGST HTS725050A7 B550  /dev/sdb

As we can continue to probe the system with the lssci tool, we see that /dev/sdb is detected as ATA. So why is Smartmontools not detecting the proper device type?

Auto-Detecting Device Types

As we discussed earlier, the SCSI command set is often times used for more than hard drives. USB and RAID devices are commonly configured as SCSI devices regardless of the storage hardware they are connecting to on the system.

Within the Linux kernel itself, we can see above that the SCSI mid layer is responsible for many device types and handles the translation to low-level drivers including libata (for ATA hardware).

Smartmontools uses a few techniques to detect the underlying storage type behind these generic block storage devices. One is by issuing a SCSI INQUIRY command to the host device and parsing the result. According to the T10 Specification, an ATA device should respond with a vendor identification of 'ATA     '  (ATA in an 8 byte frame).

Using our previous /dev/sdb and the sg_inq tool, we can see that this is indeed the case:

$ sg_inq /dev/sdb | grep 'Vendor identification'
 Vendor identification: ATA

Smartmontools will also detect this and use SCSI/ATA Translation (SAT) to communicate through the SCSI application layer directly to the ATA device. Under the hood, this is done using the autodetect_open methods. Unfortunately, the smartctl --scan does not go down this code path so will not always report on the proper device type. Running smartctl -a /dev/sdb does auto detect the device type and will give you a more accurate understanding of the actual hardware configuration.

Understanding Passthrough Devices

There are other instances of a storage device not being directly accessible to the system. For instance, a hard drive may be attached through a USB bridge, such as with an external hard drive or USB drive. We have seen previously that these USB devices can be expressed as generic SCSI devices through the kernel drivers. In these cases, all SCSI commands for querying S.M.A.R.T. data must be passed through the controller on the USB bridge to the hard drive. In order to query the S.M.A.R.T data of the device, the request must go through SAT or SCSI / NVME Translation.

Another instance requiring custom passthrough commands are for RAID controllers. For example, one class of RAID controllers which are supported by Smartmontools is the MegaRAID. Again, MegaRAID devices will be exposed as a block device on the Linux system through the sd kernel module similar to the following:

$ udevadm info -a -n /dev/sda | grep -E 'looking|DRIVER'
  looking at device '/devices/pci0000:00/0000:00:07.0/0000:06:00.0/host0/target0:2:0/0:2:0:0/block/sda':
    DRIVER==""
  looking at parent device '/devices/pci0000:00/0000:00:07.0/0000:06:00.0/host0/target0:2:0/0:2:0:0':
    DRIVERS=="sd"
  looking at parent device '/devices/pci0000:00/0000:00:07.0/0000:06:00.0/host0/target0:2:0':
    DRIVERS==""
  looking at parent device '/devices/pci0000:00/0000:00:07.0/0000:06:00.0/host0':
    DRIVERS==""
  looking at parent device '/devices/pci0000:00/0000:00:07.0/0000:06:00.0':
    DRIVERS=="megaraid_sas"
  looking at parent device '/devices/pci0000:00/0000:00:07.0':
    DRIVERS=="pcieport"
  looking at parent device '/devices/pci0000:00':
    DRIVERS==""

In order to gain information about the underlying storage behind a RAID device, Smartmontools must issue passthrough commands to the adapter. This is done using special IOCTL system calls which will be interpreted by the kernel driver and will be passed along to the firmware. The IOCTL command consists of a specially constructed packet which is sent to the kernel driver for processing. This packet will contain the necessary passthrough command fields such as:

• SCSI Command
• SCSI Command Data
• Target device ID

Since IOCTLs are not standard across all device drivers or RAID controllers, Smartmontools must create custom device classes to support all of the different ways to passthrough commands to the storage hardware.

Conclusion

We've taken a look at how Smartmontools operates and how it is able to query the different device types. We have also taken a deep dive into the different device types and how they differ between each other. In addition, the Linux storage subsystem was explored to get an idea of how devices are exposed through the Operating System. Finally, we took a look at home Smartmontools issues translation and passthrough commands to communicate with storage behind many different bridges and controllers.

This is a story about how a house holy was cut in twine. When an idea where beautiful is better than ugly goes too far. Today we are talking about the transition from Python 2 to Python 3 and the war that ensued.

Python History 201

In April of 2006, a decision was made to push a major release to the Python programming language. At the time, Python 2.5 was the latest stable release available. It was decided that after the next release of 2.6, the language would go through a major transition where core functionality would change and not be backwards compatible with prior versions. This new major release would be called Python 3.

Guido van Rossum, the author and authority of Python, layed out a timeline in which the Python 2 and prior codebase would be maintained. It was considered that in 2013 the community would embrace the new transition and the prior versions would no longer be supported. As we know now, this timeline would stretch for much further proving just how accurate us programmers are at estimating time.

The changes for Python 3 were so great, that the Python Enhanced Proposals, also known as PEPs for short, would be incremented to PEP-3000 to indicate the new version of the language. In the next sections, we will look at the modifications to the language, how this could break code written for Python 2, and how to migrate existing codebases to Python 3.

Changes in Python 3

There should be one-- and preferably only one --obvious way to do it

One of the main scriptures followed by Pythonistas is The Zen of Python. In it, a particular pragma states that there should be only one clear and concise way of doing something within the language. Because of the growth of the Python, there had become many ways of using packages and syntaxes which did not follow standards set by other similar functionalities in the language. Let's take a look at how Python 3 rectified the inconsistencies but at the cost of incompatibility with older syntaxes in the language.

Print is Now a Function

print is the only application-level functionality that has a statement dedicated to it.

Prior to the change outlined in PEP-3105, print was a language statement. With Python 3, print would now be considered a builtin function. According to the PEP, the print statement was an exception to the rule and Guido regretted this particular construct in the language.

Changing print to a function lead to a few differences which broke backwards compatibility.

>>> print('Python', '2')
('Python', '2')

For Python 2, this call is the equivalent to writing print tuple('Python', '2') or in other words: printing a tuple collection type containing the two strings 'Python' and '2' to the builtin function print.

>>> print('Python', '3')
Python 3

For Python 3, since print was converted to a function, the call above results in passing two strings as arguments to the print function.

The differences are subtle but this did result in breaking functionality for some programs.

Dictionary Keys and Values

The interface used for looping over dictionary items was also changed for Python 3. Prior, there were two redundant sets of ways to iterate over a dictionary and its elements using these methods:

• dict.keys() and dict.iterkeys()
• dict.values() and dict.itervalues()
• dict.items() and dict.iteritems()

keys, values, and items return a list type. While the iterkeys, itervalues, and iteritems methods return an iterator type. Otherwise, the two sets of methods serve the same purpose and could be used to get the same data. For this reason, it was decided to remove iterkeys, itervalues, and iteritems methods and only support keys, values, and items.

In addition, the data types returned from keys, values, and items were changed to a lightweight set equivalent type. This allows for direct comparison of results and also removes the unnecessary copying that was done internally in Python 2.

This leads to some differences as can be seen below.

>>> {'python': 2}.items()[0]
('python', 2)

In this Python 2 example, we can see that we can index into the return of the items call due to it being a list type.

>>> {'python': 3}.items()[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'dict_items' object does not support indexing

But, in the Python 3 example above, we cannot index into the result due to the change in the data type returned.

PEP-3106 covers these changes in detail.

Reorganization of the Standard Library

The next breaking change encompassed within Python 3 were the efforts to normalize the modules of the standard library. This was evaluated in PEP-3108.

Just like the language itself, Python's standard library (stdlib) has grown over the years to be very rich. But over time some modules have lost their need to be included with Python. There has also been an introduction of a naming convention for modules since Python's inception that not all modules follow.

Python has been around for a long time. Many do not realize that Python is older than many other popular programming languages such as Java, Javascript, and Ruby. In that time, standards such as naming conventions changed and the migration to Python 3 was used to normalize some of the inconsistencies in naming within the standard library.

One of the changes that has affected many Python processes was the renaming of the urllib package and its contents. The urllib package holds functionality to create HTTP requests, call HTTP endpoints, and parse HTTP responses.

Other modules renamed included:

• html
• http
• tkinter
• xmlrpc

Raising and Catching Exceptions

PEP-3109 and PEP-3110 contain the details for the changes to syntax around raising and catching exceptions in Python 3.

In Python 2, there were several ways to raise exceptions:

raise Exception, 'blah' # Python 2
raise Exception('blah') # Python 2 and 3

This was another case where there was duplicated functionality which accomplished the same result. It was proposed to remove the first syntax and keep the second when raising exceptions in Python 3.

Similarly, catching exceptions had multiple syntaxes which were equivalent.

try:
    ...
except Exception, e: # Python 2
    ...
   
try:
    ...
except Exception as e: # Python 2 and 3
    ...

Again, in Python 3 the first syntax is no longer valid and the latter was kept.

Bytes versus Strings

Prior to Python 3, byte and string data types were used interchangeably. This was due to the fact that the default type for string literals in the language was the bytestring type.

'blah' == b'blah' # Only returns True before Python 3

The problem with this functionality is that byte objects do not contain encoding information. So when converting a byte array to a unicode string, the runtime cannot determine the appropriate character set to use automatically. Thus, the language should not allow conversion freely between the types without having the option to set the encoding schema by the programmer.

In Python 3, it was decided to separate bytes and strings. String literals are now considered proper unicode strings with UTF-8 encoding by default. To convert between bytes and strings, you may use encode and decode where the encoding can be changed.

This was a step in the right direction for providing proper unicode support for the language but obviously came at the expense of breaking a subset of existing code. The enhancement was proposed as a part of PEP-358.

Converting from Python 2 to Python 3

As we have seen, the change set for Python 3 contained a lot of syntactical and functional modifications to the language. Many of these changes could break processes written with only Python 2 syntax and packages in mind. So how does one migrate to Python 3? And how does a maintainer support both versions?

The general framework to accomplishing a migration to Python 3 can be followed.

• Migrate to the latest Python 2.7 release.
• Use unit testing with sufficient code coverage to test all points in the code.
• Enable logging and watch for warnings around deprecated functions.
• Use the __future__ module in order to support both Python 2 and Python 3.

An important module to consider, during version migration and version compatibility support, is the __future__ module. Outlined in PEP-236, importing one of the future statements allows a programmer to introduce the new syntax of newer versions of the core language into older versions of the language. The benefit of using this construct is that code can be updated to new syntax one file at a time as well as maintain support for both versions of the language.

# From python-future.org - Quick Start Guide:
from __future__ import (absolute_import, division,
                        print_function, unicode_literals)
from builtins import *

Obviously, using future statements will not change your code to the new syntax. If you are looking for a more automated method of converting to Python 3, look into the 2to3 translation script. With this, Python source files are passed in and a series of fixers are applied to convert syntax in place.

Another important feature to enable during unit testing is setting the -3 flag during execution. Setting the python -3 flag enables the outputting of specific deprecation warnings to be visible during execution of code.

Dropping Support for Python 2

We have seen how the many syntax changes in Python 3 have caused compatibility issues between the older versions of the language. And we have seen some of the ways in which we can support Python 2 and 3 as package maintainers. Unfortunately, at the scale in which Python is growing, the Python collective are unable to continue to support both versions forever and at some time, Python 2 support must be dropped.

The end of life of Python 2.7 was officially extended until 2020. After, we will see many popular Python communities dropping support. Many projects are already tracking this timeline and planning the EOL of Python 2:

• Django 2.0 no longer supports Python 2
• tox is tracking an issue for dropping Python 2 support
• Using pip in a Python 2 environment emits a deprecation warning

Improvements in Python 3

We have gone over many of the problems with the migration from Python 2 to 3. But is it worth the effort to do so? Take a look at some of the new functionalities available in Python 3. Some will certainly change your mind.

• The new fstring syntax.
• The new multiprocessing packages.
• Addition of asynchronous coroutines in the asyncio package.
• Support for type hinting which derived static code analyzers such as mypy.
• Dictionaries preserving order.
• The new super style for class inheritance.
• The Abstract Base Class package.

Proton is a compatibility tool which allows running Windows only games from within a Linux desktop. The reason for releasing and supporting this type of tool is unclear. Linux use is less than 1% according to the Steam Hardware & Software Survey. That being said, it is always good to increase your marketability.

Proton was very well received within the Linux community. Under the hood, Proton integrates Wine, DXVK, and FNA in order to emulate the Windows APIs commonly used by many game engines. This compatibility is still very much in its infancy and users are encouraged to report any bugs through the Proton Issues on their GitHub.

This is a beta-only feature available by enabling the beta client. To do so, simply go to your Account Settings in Steam and enable to Beta participation.

The usage within the Steam client works pretty well. Games that are Windows only will be now be visible within the Steam Library. Within a games settings, specific Proton versions can be selected.

Compatibility

Compatibility for games varies. The Steam Issues tracker is a good place to look whether a particular game has any issues.

Skyrim

I was able to run Skyrim SE reasonably well out of the box. I did run into some issues with sound which required manual intervention. With the default FAudio libraries shipped with Proton 4.2 and prior, the NPC voices do not work. This is described in the The Elder Scrolls V: Skyrim Special Edition (489830) issue. The work around is to compile a more recent version of FAudio and override what currently resides in the Proton folder.

git clone git://github.com/FNA-XNA/FAudio.git
cd FAudio; mkdir build; cd build
cmake .. -DXNASONG=OFF -DFFMPEG=ON
make -j4
cp libFAudio.so.0 ~/.local/share/Steam/SteamApps/common/Proton\ 4.2/dist/lib64/libFAudio.so.0

This needs to be done everytime Proton is updated since the custom library will be overwritten on update. With an updated FAudio library in place, I am able to hear the NPC voices and found no other problems running Skyrim.

Conclusion

This is an exciting time to be in the Linux ecosystem. More and more companies are coming together and supporting our beloved kernel. Since it is in its infancy, Proton will have some issues. But the Linux community is great at coming together and documenting and supporting open source initiatives like this.

With the growing use of cloud technologies by companies and individuals, it is becoming ever important to understand and implement enhanced security measures for serving web content. This is necessary to not only protect yourself and your companies intellectual property, but your users and their data as well. In this post, we will look at some of the common best practices and tools to keep your website safe from the newest vulnerabilities and attack surfaces. In doing so, you will enjoy peace of mind that your content is optimized to deliver the best experience possible.

HTTPS - Use it, Enforce it

HTTPS, or HTTP over TLS, enables an encrypted connection between a client and the remote server. The security benefits are that any data transferred over the bidirectional connection will not be vulnerable to snooping or man in the middle attacks. Enabling website encryption is required for proper PCI compliance of a company. So depending on the data you are collecting from users, you must enable encryption to be up to code with regulations in your place of residence.

It's Never Been Easier

Setting up TLS, or SSL as known by its predecessor, has never been easier as many automated solutions exist for managing and auto-renewing certificates. For example, Let's Encrypt, AWS Certificate Manager, and offerings from Cloudflare exist to help manage the creation of certificates for you automatically.

SEO

There are secondary benefits to using HTTPS as well. Google has announced that secure websites will receive higher page rank than their insecure counterparts. Your site will receive less traffic, will be trusted less, and will be referred to less compared with similar sites with end-to-end encryption enforced. As a marketing agency, your content will be diminished by not using encryption as web security is a prominent factor in determining SEO.

Performance Benefits

In addition, with the advent of HTTP2 in which all browsers requires SSL to be enabled to use, there are now performance benefits for enabling encryption. HTTP2 offers multiplexing of connections to offer faster and more efficient load times compared with version one of HTTP.

Web Server Encryption

Some encryption is stronger than others. As computational power increases, a need for better encryption techniques has presented itself. In this section, we will look at tools and methods for making sure your website is up to date with the latest encryption ciphers and algorithms.

Encryption Ciphers

Cipher suites are the algorithms which are used during initial communication and play a vital role in the overall ranking of strength of encryption. As such, it is important to always update the libraries and processes used for web hosting, such as Nginx and OpenSSL, in order to take advantage of newer algorithms.

A web server can decide which ciphers to allow communication using in its configuration. That being said, a balance must be struck between allowing less strong ciphers as not all browsers and platforms support the latest and greatest techniques. Security/Server Side TLS by Mozilla is a great article on cipher suites and which to choose based on compatibility with different browsers and devices.

A modern profile might consider the follow:

Ciphersuites: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
Versions: TLSv1.2
TLS curves: prime256v1, secp384r1, secp521r1
Certificate type: ECDSA
Certificate curve: prime256v1, secp384r1, secp521r1
Certificate signature: sha256WithRSAEncryption, ecdsa-with-SHA256, ecdsa-with-SHA384, ecdsa-with-SHA512
RSA key size: 2048 (if not ecdsa)
DH Parameter size: None (disabled entirely)
ECDH Parameter size: 256
HSTS: max-age=15768000
Certificate switching: None

Securing Content

End-to-end encryption using TLS allows a secure channel between the user and server, but how do we ensure that the content and third party included on our website are what we expect? In this section we will look at more advanced techniques to ensure content injection and hijacking does not take place on our web services.

HTTP Headers

There exists a subset of HTTP headers a web administrator can use to help prevent unauthorized usage of web content. These headers are used by the browser to hint at what is to be expected on a website and any deviation of the rules should not be loaded. In this section, we will look at these headers and how they can be used to help secure your website.

Content Security Policy

The Content-Security-Policy header can be used to prevent Cross-Site Scripting attacks. These attacks inject external scripts or pages onto your site in order to gather sensitive information from users. The CSP header allows defining where content, images, and scripts should be allowed to load from on a given webpage. In addition, a uri can be defined so that any violations will be reported to the operator.

Content-Security-Policy: default-src 'self'; report-uri http://reportcollector.example.com/collector.cgi

Frame Options

The X-Frame-Options header is used to determine whether your webpage can be embedded in another webpage. A common phishing technique is to use an iframe to load another website within the parent. Thus making it look like another website allowing the attacker to gather sensitive information. This header will tell a browser which domains are allowed to embed or not allow it at all.

X-Frame-Options: allow-from https://example.com/

XSS Protection

The X-XSS-Protection header is similar to Content Security Policy which helps prevent cross-site attacks. It determines what actions to do when an attack is found from the context of a web browser.

X-XSS-Protection: 1; mode=block

Referrer Policy

The Referrer-Policy header can be used to restrict what information is sent from a browser when leaving your website. A web browser will include the full url of the website it had just visited prior to navigating to any other page on the web. With this header, you can ask to restrict the conditions and values the browser should send this information.

Referrer-Policy: no-referrer-when-downgrade

Online Domain Analyzers and Tools

The following tools are invaluable for a web hosting platform to check the ranking and correctness of the security being used.

• The SSLLabs analyzer offers an extensive free utility to test the encryption rating of a domain.
• The Security Headers analyzer can be used to verify a domains usage of HTTP security headers.
• Mozilla SSL Configuration Generator can be used to generate configuration based on several profiles of browser support for many different web servers.
• SSLShopper ssl checker is a utility which checks the correctness of ordering of a domains certificate chain.
• Sectigo crt.sh provides a way of historically searching for certificates provided by certificate authorities.

Subresource Integrity

The final security attribute we will talk about today is the Subresource Integrity. This HTML attribute allows us to provide a hash on certain HTML elements, such as script and style tags. Doing so will tell the browser that the content loaded from the resource should match what is expected on the page. Any deviation will prevent the browser from including the content on the page thus adding an additional layer of protection for our users.

<script src="https://example.com/example-framework.js"
        integrity="sha384-Li9vy3DqF8tnTXuiaAJuML3ky+er10rcgNR/VqsVpcw+ThHmYcwiB1pbOxEbzJr7"
        crossorigin="anonymous"></script>

Conclusion

Hopefully now we have learned how to make our website more secure and our users safer. There is a sense in pride when using the above tools and all of the validation tests have turned green showing our success. Even if our users are none the wiser, we can pat ourselves on the back knowing that we are taking the right steps to making the internet a safer place.

One of the most popular libraries in the Python ecosystem is the infamous requests library. Requests is used primarily for creating, sending, and parsing HTTP requests and responses. Requests tag line is "HTTP for Humans" which is very appropriate for this easy to use and wholesome project.

You will find many clients written in Python utilizing this library for communication with HTTP RESTful API services. In this post, we will look at the library and talk about some common scenarios when dealing with web APIs and how to solve them cleanly with the library. We will look at basic and more advanced usages of the library which will help you write concise pythonic code.

The Basics

The top level module exposes several functions which correspond to the standard set of HTTP verbs.

import requests
result = requests.get('https://httpbin.org/get', params={'search':'this is a search term'})
assert 'search' in result.json()['args']

This is the most basic functionality of the library in which you can send HTTP requests to a given URL and a HTTP response object is returned with attributes such as the status code, headers, cookies, and response data easily accessible.

If your code only utilizes these functions, you will want to stay tuned for the more advanced usages as they can really step up your HTTP game.

The Session

When accessing a standardized web API, it is generally advised to use a Session context object. With a session created, all subsequent requests called from within the session benefit from HTTP persistence as well as shared headers and cookies among other functionality.

For example, if an API requires all requests to have a certain HTTP header set, a session will allow you to define this default behaviour.

from requests import Session

session = Session()
session.headers.update({'X-Custom-Header': 'any value'})

result = session.get('https://httpbin.org/headers')
data = result.json()

assert 'X-Custom-Header' in data['headers']
assert data['headers']['X-Custom-Header'] == 'any value'

Event Hooks

Another important, but often overlooked functionality within requests, is the ability to register event hooks. With event hooks, we can transform the HTTP response and its data before it reaches our client code.

Also, event hooks allows us to write common error handling routines in a much simpler fashion. For example, we can catch certain status codes in an event hook and handle the error within. This is a really powerful concept since the validation will automatically be applicable to all requests sent from the session.

import requests

def forbidden_error_handler(response, *args, **kwargs):
    if response.status_code == requests.codes.forbidden:
        raise Exception('Raise some custom exception')

session = requests.Session()
session.hooks['response'].append(forbidden_error_handler)

session.get('https://httpbin.org/status/{}'.format(requests.codes.forbidden))

Authentication

There are several protocols for authenticating with HTTP services using requests.

Basic auth is a standardized authentication method supported natively in which an encoded value, given a username and password, is set in the headers of the requests.

from requests import Session
from requests.auth import HTTPBasicAuth

session = Session()
session.auth = HTTPBasicAuth('username', 'password')

result = session.get('https://httpbin.org/headers')
result_data = result.json()
assert 'Authorization' in result_data['headers']
assert result_data['headers']['Authorization'].startswith('Basic')

After registering the authentication method, every request will initiate the callable with the request. The request is then modified with the necessary authentication before transport.

Token Based Authentication

Some APIs require registering with a specific endpoint to obtain an authentication token. This token is used for subsequent calls to the API. An example of this can be found in the Django REST framework in TokenAuthentication.

If this token has a time to live, you may need to write your own authentication. For custom authentication, the AuthBase can be subclassed directly to define your own authentication.

A good example to consider is the HTTPDigestAuth class. This authentication class registers authentication specific event hooks prior to sending the request. The hooks determine if the status code of the response is that of one missing authentication. If so, an auth token is generated and the request is re-sent.

Unfortunately, this particular use case requires a good bit of logic due to the complexity of request data that can be sent with the library. For example, it must roll back iterables and file descriptors before re-sending the request. Based on a particular use case, you may be able to trim down your custom authentication class considerably.

Transport Adapters

Not for the faint of heart, Transport Adapters are used for modifying the underlying connection engine within the requests library. The adapters can be mounted to a session in order to supply specific functionality to a particular set of protocols, domains, or routes.

One common use case for supplying an adaptor is for automatic retry logic. The Retry class from urllib3 can be used to specify this information.

import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

retry_policy = Retry(3, status_forcelist=[requests.codes.server_error])
adapter = HTTPAdapter(max_retries=retry_policy)

session = requests.Session()
session.mount('https://httpbin.org/status/', adapter)

url = 'https://httpbin.org/status/{}'.format(requests.codes.server_error)

try:
    result = session.get(url)
except requests.exceptions.RetryError as e:
    print('Retries exceeded. Success!')

Conclusion

As can be seen, Requests has much more functionality than meets the eye. I hope this deep dive has given you an idea of how you can improve your uses of the library to write more clean and modular code.