A few months ago, Tesla released there updated Smart Summon feature, called Actually Smart Summon. This feature allows you to summon your car from a parking spot to your location using the Tesla app. For totally logical reasons, the API for this feature is not documented. But that doesn’t mean it’s not possible to reverse engineer it. And it also gave some insights into the workings of the Sentry mode live feed.

This is for educational purposes only. Any damage caused to the vehicle or any other object is your own responsibility. Code should not be used to break the law or violate the terms of service of Tesla Inc. This project is not affiliated with Tesla Inc. and is not endorsed by them. Code is using undocumented APIs and can break at any time.

Reverse engineering the Tesla mobile app

The first step into deciphering the Summon API is to dive into the Tesla mobile app. Since Tesla provides the same app for both Android and iOS, the codebase is written in React Native. React itself is a JavaScript library for building interactive user interfaces developed by Meta. React Native is a variant of React that allows you to build mobile applications using JavaScript. Because Android and iOS don’t support JavaScript directly, the code needs to be run on a virtual machine. React Native therefore uses its own Hermes VM. When building the app, the JavaScript code is compiled into bytecode that can be run on the Hermes VM. Because the code is compiled to some form of bytecode, it’s not directly readable. However, there are tools available to decompile this bytecode back into some form of JavaScript. hermes-dec is one of these tools but in my opinion it is still not very readable. Every new version of the Hermes VM comes with a new version of the bytecode, so the decompiler needs to be updated to support the new version. While trying to decompile the app, the decompiler crashed on some parts of the code. By just adding some try-catch blocks around the code, I was able to decompile the whole app.

    r3 = new Array(4);
    r4 = 'autopark:cmd_find_me';
    r3[0] = r4;
    r4 = 'autopark:cmd_forward';
    r3[1] = r4;
    r4 = 'autopark:cmd_reverse';
    r3[2] = r4;
    r4 = 'autopark:cmd_abort';
    r3[3] = r4;
    var _closure1_slot6 = r3;
    r3 = function() { // Environment: r1
        r0 = function(a0, a1) { // Original name: SummonCommandRTT, environment: r4
            r2 = this;
            r0 = _closure1_slot2;
            r3 = r0.default;
            r1 = _closure2_slot0;
            r0 = undefined;
            r1 = r3.bind(r0)(r2, r1);
            r1 = a0;
            r2['rttType'] = r1;
            r1 = a1;
            r2['startTime'] = r1;
            return r0;
        };

While the decompiled code is not very readable, and is quite large (around 300MB of code), it contains all the required information for the Summon API.

Beside I decompiled version, the Android app can be run in an emulator and since the app is not using certificate pinning, the traffic can be intercepted. One issue with the Android emulator is that it doesn’t support Bluetooth out of the box. To be able to connect the app to the car, the app needs to be able to connect to the car via Bluetooth to be added as a trusted device key. Using Google Bumble, a Python Bluetooth stack, the emulator can use the Bluetooth of the host. Because Bumble implements the whole Bluetooth stack, it takes full control of the Bluetooth adapter and the host can’t use it anymore. Also it requires a quite new Android version in the emulator, since older versions don’t support Bumble completely.

The first observation was that the Tesla app setups a WebSocket connection to a Tesla server. While over the WebSocket connection, Protocol Buffer encoded messages are sent. Most likely, signed commands for the car, since most of the messages contained large byte arrays with unreadable data. Not completely unexpected, since the Tesla Vehicle Command SDK also has definition files for Protocol Buffers. And after some googling, it turned out that this is the Owner/Signaling/Hermes API as used by the Tesla app.

Owner API and Hermes endpoint

While Tesla provides a public API that allows sending signed commands to the car, the Tesla app itself uses an undocumented and private API. This API, called the Owner API is, doesn’t use a RESTful API, but keeps a WebSocket connection open. Luckily, the protocol has been reverse engineered, discussed and implemented by Lothar Bach in his fork of the Tesla Vehicle Command SDK. This fork of the Tesla Vehicle Command SDK provides an updated tesla-http-proxy that creates a HTTP server that uses the Hermes endpoint to send commands to the car. The commands are encoded using protocol buffers and are the same ones as used over Bluetooth when the Tesla app controls the car from close range. And also the same commands that can be used with the signed-command endpoint of the public API.

Stream session

When looking at Summon implementation and the Sentry live feed code, it looked however like some JSON data was being sent to the car. I couldn’t find any reference to convert this to protocol buffer data and neither could I found defenitions in the Protocol Buffers definition files related to Summon or Sentry. Also I couldn’t find it in the intercepted traffic, but that was most likely due the encryption. But looking at the protocol buffers definition file car_protocol.proto and what looked like compile protocol buffer message code in the Hermes code, it became clear that some commands where missing from the official SDK. In the decompiled Tesla app, there were two additional commands defined, CreateStreamSession and StreamMessage. The definition could be filtered out of the decompiled code and added to the Tesla Vehicle Command SDK.

And after executing the CreateStreamSession command, the Tesla server responds with a STREAMING_CONFIG message and some other undecipherable messages.

{
    "expiration": 1734129437,
    "connection_timeout": 180000,
    "ping_frequency": 60000,
    "autopark": {
        "heartbeat_frequency": 100,
        "autopark_pause_timeout": 2000,
        "autopark_stop_timeout": 10000
    },
    "ice_servers": [
        { "urls": ["stun:54.73.5.101:3478", "stun:99.80.166.6:3478"] },
        {
            "urls": ["turn:52.213.90.203:3478", "turn:54.154.137.103:3478", "turn:turn6.euw1.vn.cloud.tesla.com:3478"],
            "username":"1234567890:ABCDEFG0HIJK1LMNOPQ2RSTU3VWX4YZ5",
            "credential":"AbCD0EFg/123hi4jKlMn/OPq/rs="
        }
    ]
}

It contains some timing information, and a list of ICE servers. The ICE servers suggests that WebRTC is being used. Which makes sense, since the Tesla app provides a live feed of the car’s cameras when you summon the car.

So the next step was to send a streaming message to the car. To be able to send and receive messages, I modified the tesla-http-proxy and added a WebSocket endpoint. This endpoint can be used from a browser to send and receive messages and makes it easier to create a WebRTC connection. First problem was that the tesla-http-proxy expects the client to provide a token as a header. But since WebSocket connections from the browser don’t allow custom header, I added a parameter to the command itself so the client doesn’t need to provide a token. Only problem is that you now have to restart the proxy every 8 hours, because of the token expiration and I didn’t bother to implement a refresh token logic.

So the first message I sent was { "msg_type": "control:ping" }. Resulting in a protocol buffer message that the StreamMessage command was successful. Followed by another, which were dropped by the tesla-http-proxy. After some trial and error, the StreamMessage messages seems to be sent as follow up to the CreateStreamSession command. This is relevant as the Request ID is needed to decrypt the message. The Tesla Vehicle Command SDK required some modifications to support follow up messages. After fixing, it became clear after the CreateStreamSession command, the car sends a bunch of messages related to the Sentry live feed availability and the Smart Summon status. So finally, I was able to sent messages and receive them over the WebSocket connection. To keep the connection alive, as expected by the STREAMING_CONFIG message, a ping message needs to be sent every 60 seconds. Having a connection, the next step was trying to get the live feed from the car.

Spy on the car with WebRTC

For the WebRTC connection, there are two message types “webcam:signal” and “webcam:signal_response”. The API pretty much follows the WebRTC API and was quite easy to implement. In the signal message you can either give an offer or candidate as “session_description”. The difference is made by the “type” field, which contain “candidate” for providing candidates. The response message contains the types “answer” and “candidate”.

While relatively straight forward, it took me a while to get the live feed working. The car will not accept a video stream itself and you must not forget to set the offerToReceiveVideo to true. Receisting audio or data channel will result in a failed connection.

And still, no video was sent. After looking again at the Tesla app code, I saw a function that made some strange changes to the SDP. The code ensured there was a RTP payload type 100 for VP8/90000 with some additional parameters. After adding this code for changing the SDP, video was sent but the screen remained black. And it stayed black for a day, since I forgot to stop the feed, resulting in hitting the daily limit of live feed time set by Tesla.

Figuring out that this was propably some kind of hack, I switched from Firefox to Chrome and the video feed was finally working. Additionally I implemented the “webcam:switch” message, which allows to switch between the different cameras.

< { "msg_type": "webcam:ready" }
> { "msg_type": "webcam:signal", "ctx": "sentry", "session_description": WEBRTC_OFFER }
> { "msg_type": "webcam:signal_response", "type": "candidate", "session_description": CANDIDATE }
< { "msg_type": "webcam:signal", "type": "answer", "session_description": WEBRTC_ANSWER }
< { "msg_type": "webcam:signal_response", "type": "candidate", "session_description": CANDIDATE }
> { "msg_type": "webcam:switch", "camera": "front", "prevCamera": "grid" }

Summon the Tesla

With the release of Actually Smart Summon, which also required an update for the Tesla App, I excpected a lot of changes to the API. In the early days when Smart Summon was released, you could directly connect to Tesla servers via WebSocket and send and receive JSON messages. So when I searched for “autopark:cmd_forward”, I found some older projects from before 2019. In 2019 disabled the WebSocket connection and propably started to use the Hermes endpoint with signed and encrypted commands from the Tesla app. Quite an improvement in security, to prevent some movie scenarios where someone takes over all Tesla cars. However, the commands are pretty much the same before. An “autopark:status” message is sent to give the current status of the Smart Summon functionality. Like if it’s on a public road or if the FSD computer is booting up.

Via “vehicle_data:location” you can get the current location of the car and if it’s moving. To start driving you can send a “autopark:cmd_forward”, “autopark:cmd_reverse” or “autopark:cmd_find_me” message. In case of the cmd_find_me a destination is needed. To keep the car driving, you have to sent a heartbeat, with the “autopark:heartbeat_app” message. Otherwise the error “No heartbeat received” is returned. When the car want to start driving, a “autopark:smart_summon_viz” message is sent containing the driving path. This can be used to visualize the path on a map.

The last requirement is to let the car know where you are. So you have to send a “autopark:device_location” message with your current location. Sadly, this is were the fun ends, because of the legal requirement in most countries to be in range of the car. While in North America you can summon the car from the other side of the parking lot, in Europe you have to be in Bluetooth range. Since I only setup a internet connection to the car, no Bluetooth is detected. It was suggested that in earlier versions you could just place a device with the Tesla app in the car to make the API working. Propably since before 2019 the messages were not signed and therefore could not be matched to a specific key. Now the messages are signed, the Bluetooth connection must use the same key.

The Vehicle Command SDK does provide code to connect to the car via Bluetooth. So propably it just a matter of time before someone figures this out. Although it won’t make it much easier to extend the range, but that’s propably for the best.

The Code

The Proof-of-Concept code is available on GitHub in two repositories. The first repository contains the modified Tesla Vehicle Command SDK with the WebSocket endpoint. The second repository contains the code for a single page application that can be used to send and receive messages to the car.

Additional documentation about how to run the code can be found in the README files of the repositories.