OpenPilot: An Overview and the Port to the Honda Clarity: Hardware
Note: This was written with two audiences in mind, one being for the high school at which I worked on OpenPilot as a senior project, and the other being the OpenPilot community. I did my best to find a good compromise on the amount of depth this article provides.
OpenPilot is an open-source Advanced Driver Assistance System (ADAS) developed by Comma.ai that performs the functions of Lane Keep Assist (LKAS) and Adaptive Cruise Control (ACC) by using camera and radar data to identify where you are, where the lane is, and what lead car to follow. The goal of OpenPilot is to reduce driver fatigue and increase safety on the roads. Humans are inherently terrible at driving — with poor judgement, slow reaction times, limited perception and short attention spans, people commit errors all the time on the road. Studies have shown that humans make 2–3 adjustments per second while driving, a task that can quickly become fatiguing on multi-hour long road trips or while while sitting through boring rush-hour traffic. Transferring the simple yet repetitive tasks of maintaining a safe following distance with respect to a lead vehicle and staying centered within a lane to a computer with literal light-speed reaction times and an effectively infinite attention span can greatly reduce driver fatigue and prevent common accidents such as rear-ends due to cut-ins.
Driving becomes objectively safer when OpenPilot is engaged: not only does the system function as a copilot to reduce fatigue and assist in driving the car, but it also includes built-in Driver Monitoring system that uses 3D facial reconstruction to monitor the attentiveness of the human driver. OpenPilot uses camera data to analyze the posture, head tilt, field of view, and other characteristics of the driver to decide if the human is paying attention, and if not, both aural and visual warnings will be issued. The Driver Monitoring system can detect cell phone usage, sleepiness, distractedness, and many other dangerous instances of driver negligence and mitigate the situation (THE LEADING CAUSE OF CAR ACCIDENTS IS DISTRACTED DRIVING!!!). Automation can also greatly reduce traffic and shorten travel time for all, especially as more cars on the road become equipped with ADASs like OpenPilot. For more information on how this works, check out this super informative video.
OpenPilot primarily works on cars that come from the factory with ACC and LKAS functionality — this is important because it means that the vehicle already has infrastructure in place that allows for steer by wire, accelerate by wire, and brake by wire. The inclusion of ACC also means that the car is equipped with a radar unit, which OpenPilot can use to gauge the relative distance and speed of the lead car. A good way to judge if your car is supported by OpenPilot is to check for the existence of a factory-installed ADAS camera on the windshield.
In most cars, communication between the individual computers of the car is carried out over the Controller Area Network (CAN), (unless you drive a BMW or something like that, in which case your car may use less common protocols like FlexRay.) CAN is an interesting protocol — with just two wires, many Electronic Control Units (ECUs) can talk to each other. On a CAN network, each ECU is assigned an ID, and this ID will never change, so it is a good way to tell “who” sent a message. Lower CAN IDs indicate higher priority messages, and vice versa. On CAN networks, every ECU can “hear” all messages sent on the bus, but two ECUs cannot simultaneously “talk”: if two ECUs try to broadcast messages at the same time, the ECU with the higher priority (lower ID) will take precedence and the ECU with lower priority will stop broadcasting. This is important because some messages are more important than others. For example, let’s say the outside temperature sensor (an ECU) is trying to tell the screen on your dash that the temperature outside is 69°F but your Powertrain Control Module (the PCM; another ECU) is trying to turn on the check engine light because you are getting misfires in your engine. Obviously the misfire message is more important than the temperature report message, so automakers will give the PCM a lower CAN ID with respect to the temperature sensor so the PCM will take precedence. To learn more about CAN, visit the Wikipedia page.
As you probably guessed, the factory-installed camera that performs ACC and LKAS is an ECU, and it communicates with other ECUs in the car such as the PCM for acceleration and the Electric Power Steering (EPS) motor for steering. Installing OpenPilot in a vehicle involves intercepting the CAN messages between the factory ACC/LKAS camera and the car, performing selective filtration, and injecting custom messages into the data stream before passing the message on. As stated earlier, every ECU on a CAN network can “hear” all messages sent on the bus, so if we want to perform selective filtering and inject our own commands onto the bus, we cannot leave the source ECU (in this case, the factory ADAS camera) connected to the CAN network, or else the other ECUs on the network will hear both the filtered and unfiltered messages, defeating the selective filtering process and causing the OpenPilot computer and the factory ADAS camera to fight for control of the car. To get around this, we can place the factory ADAS camera on its own isolated CAN bus and selectively forward messages between it and the car main CAN bus while injecting custom messages when needed. With this configuration, the OpenPilot computer can mimic the CAN ID of the factory ADAS camera when sending steer, brake, and accelerate commands to the car so the car thinks the factory camera is sending the messages.
Isolating the factory ADAS camera on its own CAN bus would require a device to communicate with it, and this is accomplished by the Comma Panda. The Panda is based on the STM32F413 microcontroller, which gives it three independent CAN interfaces. There are two types of Panda: A White Panda with Wifi support, and a Grey Panda with a built-in high precision GPS unit. With a Panda (and a Giraffe), the direct CAN link between the car and the factory ADAS camera can be broken and the factory camera can be isolated on its own CAN network. This uses up 2 of the 3 total available CAN interfaces on the Panda. The last CAN interface is used to tap into the Radar CAN bus to get Radar data for the driving model. This Radar CAN interface is mostly used for listening — the camera sends commands to the radar unit along with important information such as vehicle speed (because physics), and the radar responds with what it “sees”.
I mentioned a Giraffe earlier — what is that? The Comma Giraffe is a piece of passive hardware that exposes the CAN buses and presents them to an OBD-II port for a Panda to interface with. If you wanted to, you could make your own Giraffe with just wires and connectors as Giraffes are open source and there is no silicon or logic inside the Giraffe. There is a specific Giraffe for each car make, for example, there is a Toyota Giraffe, an FCA Giraffe, and a Honda Giraffe (there are actually multiple Honda Giraffes because not all Hondas use the same ADAS camera connector.) When a Giraffe and a Panda are used in tandem, it becomes possible to isolate the factory ADAS camera and selectively forward messages between it and the car, building the foundation for OpenPilot to run. The Giraffes, along with the White and Grey Pandas, are considered to be V1 hardware.
In October of 2019, Comma released V2 hardware: The Black Panda, the Car Harness, and the Comma Power V2. When the V1 hardware was released, Comma wanted the hardware to be as versatile as possible, but as OpenPilot matured, a more consumer-friendly product was needed, hence the second generation of hardware. The Black Panda includes a high-precision GPS unit just like the Grey panda, but this time the antenna is built into the enclosure, removing the need for an external GPS antenna. The Black Panda also introduced comma’s new standard communication port called OBD-C, which uses the USB C physical standard to carry CAN data. Finally, the Black Panda adds a 4th CAN interface (one more than previous Pandas) for connecting to the Comma Power V2.
The Comma Power V2 plugs into the OBD-II port found under the dashboard of the car to provide a constant power source for the OpenPilot computer, and also connects to the 4th CAN interface on the Black Panda, which allows OpenPilot to use the firmware IDs of the onboard vehicle-specific ECUs to more accurately identify the car it’s installed into.
However, the most notable component of the V2 hardware is the Car Harness. While developing OpenPilot support for many major car manufacturers such as Toyota, Honda, Subaru, FCA, and more, Comma noticed that the hardware required to enable OpenPilot functionality was similar across the manufacturers: one CAN interface for the car main CAN, one CAN interface for the isolated factory ADAS camera, and one CAN interface to tap the Radar bus. Thus, a universal piece of hardware with interchangeable connectors was created to support the cars. This new approach removed the need to engineer a new Giraffe for each car make, and also reduced the physical footprint of the installation, allowing for the connections to be better concealed for a cleaner aesthetic look. The car harness also includes a relay that can physically join together the car main CAN network and the factory ADAS camera for stock (unmodified) operation of the vehicle. This relay is a safety feature: When OpenPilot is not active, the relay defaults to a physically closed state, directly connecting the factory camera to the car main CAN and making the OpenPilot hardware invisible to the car. If the hardware were to crash or lose power, the relay would again default to a closed state. This physically closed state is called Passthrough Mode as signals are directly passed between the car and the factory camera with no intervention. The relay only opens and enters what’s called Intercept Mode when the driver enables OpenPilot and the system has passed all of its internal health and safety checks. When the relay is physically open, the factory ADAS camera is isolated onto its own CAN bus and the Black Panda is performing selective forwarding and message injection to allow OpenPilot to control the car (just like with the white and grey pandas). The relay works in conjunction with stringent software-enforced safety code to ensure that the safe operation of a vehicle is never compromised.
In January of 2020, Comma released the Comma Two, which is their new OpenPilot computing platform. Besides offering improved thermal and acoustic characteristics over the outgoing EON (previous gen hardware) and Infrared LEDs for nighttime driver attention monitoring, the biggest hardware change of the Comma Two is the integration of a Panda into the unit itself. The integrated panda, called the Uno, is very similar to the Black Panda, and shares much of its code with the Black Panda. The Uno interfaces with the Car Harness via OBD-C just like the Black Panda did, so if a car is compatible with the Black Panda, it can easily be made compatible with the Uno as well.
Moving on to the Clarity: The Clarity is a Plug-In Hybrid manufactured by Honda beginning with the 2018 model year in the US. It boasts 50+ miles of pure electric driving and an additional 300+ miles of range @ 42mpg when the battery is depleted. The Clarity comes standard with Honda’s ADAS system (called Honda Sensing) and a quick inspection of the vehicle reveals the existence of a factory-installed ADAS camera on the windshield, so the Clarity is compatible with OpenPilot, right? Unfortunately the answer is no… at least with official Comma hardware and official releases of OpenPilot. The Clarity has a different CAN network topology than every other Honda sold in the US — a fundamental hardware difference that completely breaks official support for the car.
Most Honda vehicles (and all other cars supported by OpenPilot) have one CAN bus linking the factory ADAS camera to the car and one CAN bus connecting the Radar to the factory camera. This means that breaking the link between the car main CAN and factory camera only requires two CAN interfaces, and fully installing OpenPilot only requires three CAN interfaces (the last one is for tapping the Radar CAN), which is the maximum number of CAN interfaces supported by the Pandas. However, the Clarity has two CAN buses connecting the factory camera to the car in addition to the one CAN required to connect the Radar to the factory camera, which means that to properly add support for the Clarity, we would need a theoretical “Mega Panda” with at least 5 CAN interfaces (and 6 CAN interfaces if we want to support OBD-II based VIN Fingerprinting). Unfortunately this Mega Panda does not exist, so we are left with a few options:
- Use one Panda, completely disconnect the factory ADAS camera, and implement support in software (this was the old implementation)
- Use two Pandas (aka the “mega panda”) to interface with the car and implement dual-Panda support in software
- Use two Pandas to create a CAN interceptor that makes the Clarity appear as a single-CAN car to OpenPilot, negating the need for any software modification (this is the current implementation)
- Use one Panda and use Bitbanging to emulate CAN driver hardware in software. This is a WIP! It will probably only work on the Black Panda and Uno. (Abandoned for now)
So yeah. Our current implementation just completely drops the factory ADAS camera out of the system and uses all of the available CANs on a Panda to interface directly with the car. This means that for each new release of OpenPilot, we have to go in and add our Clarity code and fix whatever is broken (something breaks with pretty much every release). This is time consuming and pretty annoying to deal with. With this implementation, we also lose all stock features that the factory ADAS provides. On the Clarity, we lose Automatic Emergency Braking.
If you look carefully, you’ll see that with our current implementation, we’ve connected CAN2 on the Panda to FCAN2 on the car. This is fine if a White or Grey Panda is being used because the Panda will only write to the CAN buses if instructed to do so by software. However, if a Car Harness and Black Panda or Uno is used, a problem arises: the Car Harness relay directly connects CAN0 to CAN2. For most cars, this is great: you get all the safety benefits and seamless switching to stock discussed above. However, in the case of the Clarity, this is pretty bad: If the relay is closed, then the two main FCANs of the car are bridged together… which makes the car go completely haywire. Literally every single error light on the dash will illuminate, and many of the car systems will start to malfunction. This can be dangerous as the user may partially lose control over the vehicle. For example, when the relay is closed, the transmission will start to erratically shift to random gears by itself and ignore any user input. Currently, this problem is addressed in software: we have forced the relay to always be open under all circumstances. However, this is a software fix for a hardware flaw… a better solution needs to be found.
Option two would be the route to take if one was trying to get comma to upstream support for the Clarity and many other cars that are limited by the number of CAN interfaces available on one Panda. Two pandas could be “glued together” onto one PCB, which would give us a total of 6 CAN interfaces to work with (which would fully support the Clarity perfectly). Software infrastructure would need to be written to enable support for this new dual-panda configuration.
Option three could work: We could have one panda sit independently of openpilot, and forward messages between a “master” CAN bus and a “slave” CAN bus. This would allow us to read and write from the slave CAN bus by reading/writing just to the main CAN, and having the independent panda forward messages between the busses as needed. With this method, it could be possible to overload the main CAN bus, so more research is needed.
This article has given an overview of the hardware work needed to add OpenPilot support to the Honda Clarity: an article on the software work may come later. Stay posted!
