From beb58b4bf08266e296d4f373f9a24272691c31c6 Mon Sep 17 00:00:00 2001 From: Jannis R Date: Wed, 14 Mar 2018 20:46:10 +0100 Subject: [PATCH 01/13] docs: move list of methods to docs/index.md :memo: --- docs/index.md | 9 +++++++++ readme.md | 8 +------- 2 files changed, 10 insertions(+), 7 deletions(-) create mode 100644 docs/index.md diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 00000000..574a0c98 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,9 @@ +# API documentation + +- [`journeys(from, to, [opt])`](journeys.md) – get journeys between locations +- [`journeyLeg(ref, lineName, [opt])`](journey-leg.md) – get details for a leg of a journey +- [`departures(station, [opt])`](departures.md) – query the next departures at a station +- [`locations(query, [opt])`](locations.md) – find stations, POIs and addresses +- [`location(id)`](location.md) – get details about a location +- [`nearby(location, [opt])`](nearby.md) – show stations & POIs around +- [`radar(north, west, south, east, [opt])`](radar.md) – find all vehicles currently in a certain area diff --git a/readme.md b/readme.md index d627004a..4bcfe31a 100644 --- a/readme.md +++ b/readme.md @@ -32,13 +32,7 @@ npm install hafas-client ## API -- [`journeys(from, to, [opt])`](docs/journeys.md) – get journeys between locations -- [`journeyLeg(ref, lineName, [opt])`](docs/journey-leg.md) – get details for a leg of a journey -- [`departures(station, [opt])`](docs/departures.md) – query the next departures at a station -- [`locations(query, [opt])`](docs/locations.md) – find stations, POIs and addresses -- [`location(id)`](docs/location.md) – get details about a location -- [`nearby(location, [opt])`](docs/nearby.md) – show stations & POIs around -- [`radar(north, west, south, east, [opt])`](docs/radar.md) – find all vehicles currently in a certain area +[API documentation](docs/index.md) ## Usage From 1b367ac899808018d2f08aeb5c753dbdfa8e7451 Mon Sep 17 00:00:00 2001 From: Jannis R Date: Wed, 14 Mar 2018 22:52:36 +0100 Subject: [PATCH 02/13] writing a profile guide: part 1 --- docs/index.md | 4 +++ docs/writing-a-profile.md | 57 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+) create mode 100644 docs/writing-a-profile.md diff --git a/docs/index.md b/docs/index.md index 574a0c98..ea55aa82 100644 --- a/docs/index.md +++ b/docs/index.md @@ -7,3 +7,7 @@ - [`location(id)`](location.md) – get details about a location - [`nearby(location, [opt])`](nearby.md) – show stations & POIs around - [`radar(north, west, south, east, [opt])`](radar.md) – find all vehicles currently in a certain area + +## Writing a profile + +Check [the guide](writing-a-profile.md). diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md new file mode 100644 index 00000000..4dccaf02 --- /dev/null +++ b/docs/writing-a-profile.md @@ -0,0 +1,57 @@ +# Writing a profile + +**Per endpoint, there is an endpoint-specific customisation called *profile*** which may for example do the following: + +- handle the additional requirements of the endpoint (e.g. authentication), +- extract additional information from the data provided by the endpoint, +- guard against triggering bugs of certain endpoints (e.g. time limits). + +**This guide is about writing such a profile.** If you just want to use an already supported endpoint, refer to the [API documentation](index.md) instead. + +*Note*: You can always ask for help by creating an issue! We're motivated to help people expand the scope of this library. + +## 0. How does it work? + +A profile contains two things: + +- **mandatory details about the HAFAS endpoint** + - `endpoint`: The protocol, host and path of the endpoint. + - `locale`: The [BCP 47](https://en.wikipedia.org/wiki/IETF_language_tag) [locale](https://en.wikipedia.org/wiki/Locale_(computer_software)) of your endpoint (or the area that your endpoint covers). + - `timezone`: An [IANA-time-zone](https://www.iana.org/time-zones)-compatible [timezone](https://en.wikipedia.org/wiki/Time_zone) of your endpoint. +- **flags indicating that features are supported by the endpoint** – e.g. `journeyRef` +- **methods overriding the [default profile](../lib/default-profile.js)** + +As an example, let's say that our endpoint `https://example.org/bin/mgate.exe` has the timezone `Europe/Vienna` and locale `de-AT`. It also returns all lines names prefixed with `foo `. We can strip them like this: + +```js +// get the default line parser +const createParseLine = require('hafas-client/parse/line') + +const createParseLineWithoutFoo = (profile, operators) => { + const parseLine = createParseLine(profile, operators) + + // wrapper function with additional logic + const parseLineWithoutFoo = (l) => { + const line = parseLine(l) + line.name = line.name.replace(/foo /g, '') + return line + } + return parseLineWithoutFoo +} +``` + +Our profile will look like this: + +```js +const myProfile = { + endpoint: 'https://example.org/bin/mgate.exe', + parseLine: createParseLineWithoutFoo +} +``` + +If you pass this profile into `hafas-client`, the `parseLine` method will override [the default one](../parse/line.js). + +```js +const createClient = require('hafas-client') +const client = createClient(myProfile) // create a client with our profile +``` From a8f0d753697fa581270127fa6c0faa5f623d5edf Mon Sep 17 00:00:00 2001 From: Jannis R Date: Wed, 14 Mar 2018 23:38:35 +0100 Subject: [PATCH 03/13] writing a profile guide: part 2 --- docs/writing-a-profile.md | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index 4dccaf02..12989e0a 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -10,9 +10,9 @@ *Note*: You can always ask for help by creating an issue! We're motivated to help people expand the scope of this library. -## 0. How does it work? +## 0. How does the profiles work? -A profile contains two things: +A profile contains of three things: - **mandatory details about the HAFAS endpoint** - `endpoint`: The protocol, host and path of the endpoint. @@ -55,3 +55,35 @@ If you pass this profile into `hafas-client`, the `parseLine` method will overri const createClient = require('hafas-client') const client = createClient(myProfile) // create a client with our profile ``` + +## 1. Setup + +*Note*: There are many ways to find the required values. This way is rather easy and has worked for most of the apps that I've looked at so far. + +1. **Get an iOS or Android device and download the "official" app** for the public transport provider that you want to build a profile for. +2. **Configure a [man-in-the-middle HTTP proxy](https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/)** like [mitmproxy](https://mitmproxy.org). +3. **Record requests of the app.** + - To help others in the future, post the requests (in their entirety!) on GitHub, e.g. in a format like [this](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886). This will also let us help you if you have any questions. + - Make sure to cover all relevant sections of the app, e.g. "journeys", "departures", "live map". Better record more than less; You will regret not having enough information later on. + - *Note*: This method does not work if the app uses [public key pinning](https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning). In this case, please create an issue, so we can discuss other techniques. + +## 2. Basic profile + +- **Identify the `endpoint`.** The protocol, host and path of the endpoint, *but not* the query string. + - *Note*: **`hafas-client` for now only supports the interface providing JSON** (generated from XML), which is being used by the corresponding iOS/Android apps. It supports neither the JSONP, nor the XML, nor the HTML interface. If the endpoint does not end in `mgate.exe`, it mostly likely won't work. +- **Identify the `locale`.** Basically guess work; Use the date & time formats as an indicator. +- **Identify the `timezone`.** This may be tricky, a for example [Deutsche Bahn](https://en.wikipedia.org/wiki/Deutsche_Bahn) returns departures for Moscow as `+01:00` instead of `+03:00`. + +## 3. Additional info + +We consider these *optional* improvements: + +- **Check if the endpoint supports the journey legs call.** + - In the app, check if you can query details for the status of a single journey leg. It should load realtime delays and the current progress. + - If this feature is supported, add `journeyLeg: true` to the profile. +- **Check if the endpoint supports the live map call.** Does the app have a "live map" showing all vehicles within an area? If so, add `radar: true` to the profile. +- **Consider transforming station & line names** into the formats that's most suitable for *local users*. Some examples: + - `M13 (Tram)` -> `M13`. With Berlin context, it is obvious that `M13` is a tram. + - `Berlin Jungfernheide Bhf` -> `Berlin Jungfernheide`. With local context, it's obvious that *Jungfernheide* is a train station. +- **Check if the endpoint has non-obvious limitations** and let use know about these. Examples: + - Some endpoints have a time limit, after which they won't return more departures, but silently discard them. From 5ea6050b4fc3f1efddf274e8cd2241e90acdf7e3 Mon Sep 17 00:00:00 2001 From: Jannis R Date: Thu, 15 Mar 2018 01:04:06 +0100 Subject: [PATCH 04/13] writing a profile guide: authentication & example profile --- docs/writing-a-profile.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index 12989e0a..dfe031b8 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -51,21 +51,16 @@ const myProfile = { If you pass this profile into `hafas-client`, the `parseLine` method will override [the default one](../parse/line.js). -```js -const createClient = require('hafas-client') -const client = createClient(myProfile) // create a client with our profile -``` - ## 1. Setup *Note*: There are many ways to find the required values. This way is rather easy and has worked for most of the apps that I've looked at so far. 1. **Get an iOS or Android device and download the "official" app** for the public transport provider that you want to build a profile for. 2. **Configure a [man-in-the-middle HTTP proxy](https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/)** like [mitmproxy](https://mitmproxy.org). + - *Note*: This method does not work if the app uses [public key pinning](https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning). In this case (the app won't be able to query data), please create an issue, so we can discuss other techniques. 3. **Record requests of the app.** - To help others in the future, post the requests (in their entirety!) on GitHub, e.g. in a format like [this](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886). This will also let us help you if you have any questions. - Make sure to cover all relevant sections of the app, e.g. "journeys", "departures", "live map". Better record more than less; You will regret not having enough information later on. - - *Note*: This method does not work if the app uses [public key pinning](https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning). In this case, please create an issue, so we can discuss other techniques. ## 2. Basic profile @@ -73,6 +68,11 @@ const client = createClient(myProfile) // create a client with our profile - *Note*: **`hafas-client` for now only supports the interface providing JSON** (generated from XML), which is being used by the corresponding iOS/Android apps. It supports neither the JSONP, nor the XML, nor the HTML interface. If the endpoint does not end in `mgate.exe`, it mostly likely won't work. - **Identify the `locale`.** Basically guess work; Use the date & time formats as an indicator. - **Identify the `timezone`.** This may be tricky, a for example [Deutsche Bahn](https://en.wikipedia.org/wiki/Deutsche_Bahn) returns departures for Moscow as `+01:00` instead of `+03:00`. +- **Copy the authentication** and other meta fields, namely `ver`, `ext`, `client` and `lang`. + - You can find these fields in the root of each request JSON. Check [a VBB request](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886#file-1-http-L13-L22) and the corresponding [the VBB profile](https://github.com/derhuerst/hafas-client/blob/6e61097687a37b60d53e767f2711466b80c5142c/p/vbb/index.js#L22-L29) for an example. + - Add a function `transformReqBody(body)` to your profile, which assigns them to `body`. + +If you want, you can now **verify that the profile works**; I've prepared [a script](https://runkit.com/derhuerst/hafas-client-profile-example) for that. Alternatively, [submit Pull Request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) and I will help you out with testing and improvements. ## 3. Additional info From 01fd94f70ad6f0db69a0306c2f804eb1305e8c29 Mon Sep 17 00:00:00 2001 From: Jannis R Date: Thu, 15 Mar 2018 01:10:52 +0100 Subject: [PATCH 05/13] writing a profile guide: minor improvements --- docs/{index.md => readme.md} | 0 docs/writing-a-profile.md | 8 ++++---- readme.md | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) rename docs/{index.md => readme.md} (100%) diff --git a/docs/index.md b/docs/readme.md similarity index 100% rename from docs/index.md rename to docs/readme.md diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index dfe031b8..7f2496de 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -6,9 +6,9 @@ - extract additional information from the data provided by the endpoint, - guard against triggering bugs of certain endpoints (e.g. time limits). -**This guide is about writing such a profile.** If you just want to use an already supported endpoint, refer to the [API documentation](index.md) instead. +This guide is about writing such a profile. If you just want to use an already supported endpoint, refer to the [API documentation](readme.md) instead. -*Note*: You can always ask for help by creating an issue! We're motivated to help people expand the scope of this library. +*Note*: **If you get stuck, ask for help by creating an issue!** We're motivated to help people expand the scope of this library. ## 0. How does the profiles work? @@ -59,7 +59,7 @@ If you pass this profile into `hafas-client`, the `parseLine` method will overri 2. **Configure a [man-in-the-middle HTTP proxy](https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/)** like [mitmproxy](https://mitmproxy.org). - *Note*: This method does not work if the app uses [public key pinning](https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning). In this case (the app won't be able to query data), please create an issue, so we can discuss other techniques. 3. **Record requests of the app.** - - To help others in the future, post the requests (in their entirety!) on GitHub, e.g. in a format like [this](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886). This will also let us help you if you have any questions. + - To help others in the future, post the requests (in their entirety!) on GitHub, e.g. in as format like [this](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886). This will also let us help you if you have any questions. - Make sure to cover all relevant sections of the app, e.g. "journeys", "departures", "live map". Better record more than less; You will regret not having enough information later on. ## 2. Basic profile @@ -76,7 +76,7 @@ If you want, you can now **verify that the profile works**; I've prepared [a scr ## 3. Additional info -We consider these *optional* improvements: +We consider these improvements to be *optional*: - **Check if the endpoint supports the journey legs call.** - In the app, check if you can query details for the status of a single journey leg. It should load realtime delays and the current progress. diff --git a/readme.md b/readme.md index 4bcfe31a..f2f620e8 100644 --- a/readme.md +++ b/readme.md @@ -32,7 +32,7 @@ npm install hafas-client ## API -[API documentation](docs/index.md) +[API documentation](docs/readme.md) ## Usage From e33f5b213d85848e2b80085302e966b23c44d188 Mon Sep 17 00:00:00 2001 From: Jannis R Date: Thu, 15 Mar 2018 01:33:16 +0100 Subject: [PATCH 06/13] writing a profile guide: checksum, mic, mac --- docs/writing-a-profile.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index 7f2496de..dc206f94 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -71,6 +71,7 @@ If you pass this profile into `hafas-client`, the `parseLine` method will overri - **Copy the authentication** and other meta fields, namely `ver`, `ext`, `client` and `lang`. - You can find these fields in the root of each request JSON. Check [a VBB request](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886#file-1-http-L13-L22) and the corresponding [the VBB profile](https://github.com/derhuerst/hafas-client/blob/6e61097687a37b60d53e767f2711466b80c5142c/p/vbb/index.js#L22-L29) for an example. - Add a function `transformReqBody(body)` to your profile, which assigns them to `body`. + - Some profiles have a `checksum` parameter (like [here](https://gist.github.com/derhuerst/2a735268bd82a0a6779633f15dceba33#file-journey-details-1-http-L1)) or two `mic` & `mac` parameters (like [here](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886#file-1-http-L1)). If you see one of them in your requests, jump to [*Appendix A: checksum, mic, mac*](#appendix-a-checksum-mic-mac). Unfortunately, this is necessary to get the profile working. If you want, you can now **verify that the profile works**; I've prepared [a script](https://runkit.com/derhuerst/hafas-client-profile-example) for that. Alternatively, [submit Pull Request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) and I will help you out with testing and improvements. @@ -87,3 +88,23 @@ We consider these improvements to be *optional*: - `Berlin Jungfernheide Bhf` -> `Berlin Jungfernheide`. With local context, it's obvious that *Jungfernheide* is a train station. - **Check if the endpoint has non-obvious limitations** and let use know about these. Examples: - Some endpoints have a time limit, after which they won't return more departures, but silently discard them. + +--- + +## Appendix A: `checksum`, `mic`, `mac` + +As far as I know, there are three different types of authentication used among HAFAS deployments. + +### unprotected endpoints + +You can just query these if you send a formally correct request. + +### endpoints using the `checksum` query parameter + +`checksum` is a [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code): `hafas-client` will compute it by [hashing](https://en.wikipedia.org/wiki/Hash_function) the request body and a *salt* (which means secret). **This secret can be read from the config file inside the app bundle.** There is no guide for this yet, so please open an issue instead. + +### endpoints using the `mic` & `mac` query parameters + +`mic` is a [message integrity code](https://en.wikipedia.org/wiki/Message_authentication_code), the [hash](https://en.wikipedia.org/wiki/Hash_function) of the request body. + +`mac` is a [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code), the hash of `mic` and a *salt* (which means secret). **This secret can be read from the config file inside the app bundle.** There is no guide for this yet, so please open an issue instead. From 4533a70c4145dcc1b13c27f2bc4d2bdca2e00a0e Mon Sep 17 00:00:00 2001 From: Jannis R Date: Thu, 15 Mar 2018 01:35:01 +0100 Subject: [PATCH 07/13] writing a profile guide: minor improvements --- docs/writing-a-profile.md | 31 +++++++++++++++++-------------- lib/default-profile.js | 2 +- 2 files changed, 18 insertions(+), 15 deletions(-) diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index dc206f94..e264667b 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -10,7 +10,7 @@ This guide is about writing such a profile. If you just want to use an already s *Note*: **If you get stuck, ask for help by creating an issue!** We're motivated to help people expand the scope of this library. -## 0. How does the profiles work? +## 0. How do the profiles work? A profile contains of three things: @@ -21,7 +21,19 @@ A profile contains of three things: - **flags indicating that features are supported by the endpoint** – e.g. `journeyRef` - **methods overriding the [default profile](../lib/default-profile.js)** -As an example, let's say that our endpoint `https://example.org/bin/mgate.exe` has the timezone `Europe/Vienna` and locale `de-AT`. It also returns all lines names prefixed with `foo `. We can strip them like this: +As an example, let's say we have an [Austrian](https://en.wikipedia.org/wiki/Austria) endpoint: + +```js +const myProfile = { + endpoint: 'https://example.org/bin/mgate.exe', + locale: 'de-AT', + timezone: 'Europe/Vienna' +} +``` + +If you pass this profile into `hafas-client`, the `parseLine` method will override [the default one](../parse/line.js). + +Assuming the endpoint returns all lines names prefixed with `foo `, We can strip them like this: ```js // get the default line parser @@ -38,19 +50,10 @@ const createParseLineWithoutFoo = (profile, operators) => { } return parseLineWithoutFoo } + +profile.parseLine = createParseLineWithoutFoo ``` -Our profile will look like this: - -```js -const myProfile = { - endpoint: 'https://example.org/bin/mgate.exe', - parseLine: createParseLineWithoutFoo -} -``` - -If you pass this profile into `hafas-client`, the `parseLine` method will override [the default one](../parse/line.js). - ## 1. Setup *Note*: There are many ways to find the required values. This way is rather easy and has worked for most of the apps that I've looked at so far. @@ -69,7 +72,7 @@ If you pass this profile into `hafas-client`, the `parseLine` method will overri - **Identify the `locale`.** Basically guess work; Use the date & time formats as an indicator. - **Identify the `timezone`.** This may be tricky, a for example [Deutsche Bahn](https://en.wikipedia.org/wiki/Deutsche_Bahn) returns departures for Moscow as `+01:00` instead of `+03:00`. - **Copy the authentication** and other meta fields, namely `ver`, `ext`, `client` and `lang`. - - You can find these fields in the root of each request JSON. Check [a VBB request](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886#file-1-http-L13-L22) and the corresponding [the VBB profile](https://github.com/derhuerst/hafas-client/blob/6e61097687a37b60d53e767f2711466b80c5142c/p/vbb/index.js#L22-L29) for an example. + - You can find these fields in the root of each request JSON. Check [a VBB request](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886#file-1-http-L13-L22) and [the corresponding VBB profile](https://github.com/derhuerst/hafas-client/blob/6e61097687a37b60d53e767f2711466b80c5142c/p/vbb/index.js#L22-L29) for an example. - Add a function `transformReqBody(body)` to your profile, which assigns them to `body`. - Some profiles have a `checksum` parameter (like [here](https://gist.github.com/derhuerst/2a735268bd82a0a6779633f15dceba33#file-journey-details-1-http-L1)) or two `mic` & `mac` parameters (like [here](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886#file-1-http-L1)). If you see one of them in your requests, jump to [*Appendix A: checksum, mic, mac*](#appendix-a-checksum-mic-mac). Unfortunately, this is necessary to get the profile working. diff --git a/lib/default-profile.js b/lib/default-profile.js index cf8493d8..340de25f 100644 --- a/lib/default-profile.js +++ b/lib/default-profile.js @@ -59,7 +59,7 @@ const defaultProfile = { formatRectangle, filters, - journeysNumF: true, // `journeys()` method: support for `numF` field + journeysNumF: true, // `journeys()` method: support for `numF` field? journeyLeg: false, radar: false } From 16d4ee4fd1a9e28f68331d87034d0472583df89f Mon Sep 17 00:00:00 2001 From: Jannis R Date: Thu, 15 Mar 2018 16:57:20 +0100 Subject: [PATCH 08/13] writing a profile guide: minor improvements --- docs/writing-a-profile.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index e264667b..873e5d5f 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -8,7 +8,7 @@ This guide is about writing such a profile. If you just want to use an already supported endpoint, refer to the [API documentation](readme.md) instead. -*Note*: **If you get stuck, ask for help by creating an issue!** We're motivated to help people expand the scope of this library. +*Note*: **If you get stuck, ask for help by [creating an issue](https://github.com/derhuerst/hafas-client/issues/new)!** We're motivated to help people expand the scope of this library. ## 0. How do the profiles work? @@ -58,9 +58,11 @@ profile.parseLine = createParseLineWithoutFoo *Note*: There are many ways to find the required values. This way is rather easy and has worked for most of the apps that I've looked at so far. +[There's a video showing the following steps](https://stuff.jannisr.de/how-to-record-hafas-requests.mp4). + 1. **Get an iOS or Android device and download the "official" app** for the public transport provider that you want to build a profile for. 2. **Configure a [man-in-the-middle HTTP proxy](https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/)** like [mitmproxy](https://mitmproxy.org). - - *Note*: This method does not work if the app uses [public key pinning](https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning). In this case (the app won't be able to query data), please create an issue, so we can discuss other techniques. + - *Note*: This method does not work if the app uses [public key pinning](https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning). In this case (the app won't be able to query data), please [create an issue](https://github.com/derhuerst/hafas-client/issues/new), so we can discuss other techniques. 3. **Record requests of the app.** - To help others in the future, post the requests (in their entirety!) on GitHub, e.g. in as format like [this](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886). This will also let us help you if you have any questions. - Make sure to cover all relevant sections of the app, e.g. "journeys", "departures", "live map". Better record more than less; You will regret not having enough information later on. @@ -76,6 +78,8 @@ profile.parseLine = createParseLineWithoutFoo - Add a function `transformReqBody(body)` to your profile, which assigns them to `body`. - Some profiles have a `checksum` parameter (like [here](https://gist.github.com/derhuerst/2a735268bd82a0a6779633f15dceba33#file-journey-details-1-http-L1)) or two `mic` & `mac` parameters (like [here](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886#file-1-http-L1)). If you see one of them in your requests, jump to [*Appendix A: checksum, mic, mac*](#appendix-a-checksum-mic-mac). Unfortunately, this is necessary to get the profile working. +todo: products/modes + If you want, you can now **verify that the profile works**; I've prepared [a script](https://runkit.com/derhuerst/hafas-client-profile-example) for that. Alternatively, [submit Pull Request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) and I will help you out with testing and improvements. ## 3. Additional info @@ -104,10 +108,10 @@ You can just query these if you send a formally correct request. ### endpoints using the `checksum` query parameter -`checksum` is a [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code): `hafas-client` will compute it by [hashing](https://en.wikipedia.org/wiki/Hash_function) the request body and a *salt* (which means secret). **This secret can be read from the config file inside the app bundle.** There is no guide for this yet, so please open an issue instead. +`checksum` is a [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code): `hafas-client` will compute it by [hashing](https://en.wikipedia.org/wiki/Hash_function) the request body and a *salt* (which means secret). **This secret can be read from the config file inside the app bundle.** There is no guide for this yet, so please [open an issue](https://github.com/derhuerst/hafas-client/issues/new) instead. ### endpoints using the `mic` & `mac` query parameters `mic` is a [message integrity code](https://en.wikipedia.org/wiki/Message_authentication_code), the [hash](https://en.wikipedia.org/wiki/Hash_function) of the request body. -`mac` is a [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code), the hash of `mic` and a *salt* (which means secret). **This secret can be read from the config file inside the app bundle.** There is no guide for this yet, so please open an issue instead. +`mac` is a [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code), the hash of `mic` and a *salt* (which means secret). **This secret can be read from the config file inside the app bundle.** There is no guide for this yet, so please [open an issue](https://github.com/derhuerst/hafas-client/issues/new) instead. From 43b59961d10bff36008f8324b3c2f7ac2295b0f5 Mon Sep 17 00:00:00 2001 From: Jannis R Date: Fri, 16 Mar 2018 01:23:50 +0100 Subject: [PATCH 09/13] writing a profile guide: products part 1 --- docs/writing-a-profile.md | 56 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 54 insertions(+), 2 deletions(-) diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index 873e5d5f..235971be 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -78,11 +78,63 @@ profile.parseLine = createParseLineWithoutFoo - Add a function `transformReqBody(body)` to your profile, which assigns them to `body`. - Some profiles have a `checksum` parameter (like [here](https://gist.github.com/derhuerst/2a735268bd82a0a6779633f15dceba33#file-journey-details-1-http-L1)) or two `mic` & `mac` parameters (like [here](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886#file-1-http-L1)). If you see one of them in your requests, jump to [*Appendix A: checksum, mic, mac*](#appendix-a-checksum-mic-mac). Unfortunately, this is necessary to get the profile working. -todo: products/modes +## 3. Products + +In `hafas-client`, there's a difference between the `mode` and the `product` field: + +- The `mode` field describes the mode of transport in general. [Standardised by the *Friendly Public Transport Format* `1.0.1`](https://github.com/public-transport/friendly-public-transport-format/blob/1.0.1/spec/readme.md#modes), it is on purpose limited to a very small number of possible values, e.g. `train` or `bus`. +- The value for `product` relates to how a means of transport "works" *in local context*. Example: Even though [*S-Bahn*](https://en.wikipedia.org/wiki/Berlin_S-Bahn) and [*U-Bahn*](https://en.wikipedia.org/wiki/Berlin_U-Bahn) in Berlin are both `train`s, they have different operators, service patterns, stations and look different. Therefore, they are two distinct `product`s `subway` and `suburban`. + +**Specify `product`s that appear in the app** you recorded requests of. For a fictional transit network, this may look like this: + +```js +const products = { + commuterTrain: { + product: 'commuterTrain', + mode: 'train', + bitmask: 1, + name: 'ACME Commuter Rail', + short: 'CR' + }, + metro: { + product: 'metro', + mode: 'train', + bitmask: 2, + name: 'Foo Bar Metro', + short: 'M' + } +} +``` + +Let's break this down: + +- `product` should contain a sensible, [camelCased](https://en.wikipedia.org/wiki/Camel_case#Variations_and_synonyms), alphanumeric identifier. Use it for the key in the `products` object as well. +- `mode` should be a [valid *Friendly Public Transport Format* `1.0.1` mode](https://github.com/public-transport/friendly-public-transport-format/blob/1.0.1/spec/readme.md#modes). +- HAFAS endpoints work with a [bitmask](https://en.wikipedia.org/wiki/Mask_(computing)#Arguments_to_functions) that toggles the individual products. `bitmask` should toggle the appropriate bit(s) in the bitmask (see below). +- `name` should be a short, but distinct name for the means of transport, *just precise enough in local context*. In Berlin, `S-Bahn commuter rail` would be too much, because everyone knows what `S-Bahn` means. +- `short` short be the shortest possible symbol that identifies the product. + +todo: `defaultProducts`, `allProducts`, `bitmasks`, add to profile If you want, you can now **verify that the profile works**; I've prepared [a script](https://runkit.com/derhuerst/hafas-client-profile-example) for that. Alternatively, [submit Pull Request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) and I will help you out with testing and improvements. -## 3. Additional info +### Finding the right values for the `bitmask` field + +As shown in [the video](https://stuff.jannisr.de/how-to-record-hafas-requests.mp4), search for a journey and toggle off one product at a time, recording the requests. After extracting the products bitmask ([example](https://gist.github.com/derhuerst/193ef489f8aa50c2343f8bf1f2a22069#file-via-http-L34)) you will end up with values looking like these: + +``` +toggles value binary notation subtraction bit(s) +all products 255 11111111 255 - 0 +all but ACME Commuter Rail 127 01111111 255 - 2^7 2^7 +all but Foo Bar Metro 191 10111111 255 - 2^6 2^6 +all but product C 223 11011111 255 - 2^5 2^5 +all but product D 239 11101111 255 - 2^4 2^4 +all but product E 243 11110011 255 - 2^3 - 2^2 2^3 & 2^2 +all but product F 253 11111101 255 - 2^1 2^1 +all but product G 254 11111110 255 - 2^0 2^0 +``` + +## 4. Additional info We consider these improvements to be *optional*: From 9b6c07e477f736660c7f7cc53a59e869220eb551 Mon Sep 17 00:00:00 2001 From: Jannis R Date: Fri, 16 Mar 2018 01:43:10 +0100 Subject: [PATCH 10/13] writing a profile guide: minor improvements --- docs/writing-a-profile.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index 235971be..59b5a11c 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -8,7 +8,7 @@ This guide is about writing such a profile. If you just want to use an already supported endpoint, refer to the [API documentation](readme.md) instead. -*Note*: **If you get stuck, ask for help by [creating an issue](https://github.com/derhuerst/hafas-client/issues/new)!** We're motivated to help people expand the scope of this library. +*Note*: **If you get stuck, ask for help by [creating an issue](https://github.com/derhuerst/hafas-client/issues/new)!** I want to help people expand the scope of this library. ## 0. How do the profiles work? @@ -58,14 +58,14 @@ profile.parseLine = createParseLineWithoutFoo *Note*: There are many ways to find the required values. This way is rather easy and has worked for most of the apps that I've looked at so far. -[There's a video showing the following steps](https://stuff.jannisr.de/how-to-record-hafas-requests.mp4). - 1. **Get an iOS or Android device and download the "official" app** for the public transport provider that you want to build a profile for. 2. **Configure a [man-in-the-middle HTTP proxy](https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/)** like [mitmproxy](https://mitmproxy.org). + - Configure your device to trust the self-signed SSL certificate, [as outlined in the mitmproxy docs](https://docs.mitmproxy.org/stable/concepts-certificates/). - *Note*: This method does not work if the app uses [public key pinning](https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning). In this case (the app won't be able to query data), please [create an issue](https://github.com/derhuerst/hafas-client/issues/new), so we can discuss other techniques. 3. **Record requests of the app.** - - To help others in the future, post the requests (in their entirety!) on GitHub, e.g. in as format like [this](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886). This will also let us help you if you have any questions. + - [There's a video showing this step](https://stuff.jannisr.de/how-to-record-hafas-requests.mp4). - Make sure to cover all relevant sections of the app, e.g. "journeys", "departures", "live map". Better record more than less; You will regret not having enough information later on. + - To help others in the future, post the requests (in their entirety!) on GitHub, e.g. in as format like [this](https://gist.github.com/derhuerst/5fa86ed5aec63645e5ae37e23e555886). This will also let us help you if you have any questions. ## 2. Basic profile @@ -108,15 +108,15 @@ const products = { Let's break this down: -- `product` should contain a sensible, [camelCased](https://en.wikipedia.org/wiki/Camel_case#Variations_and_synonyms), alphanumeric identifier. Use it for the key in the `products` object as well. -- `mode` should be a [valid *Friendly Public Transport Format* `1.0.1` mode](https://github.com/public-transport/friendly-public-transport-format/blob/1.0.1/spec/readme.md#modes). -- HAFAS endpoints work with a [bitmask](https://en.wikipedia.org/wiki/Mask_(computing)#Arguments_to_functions) that toggles the individual products. `bitmask` should toggle the appropriate bit(s) in the bitmask (see below). -- `name` should be a short, but distinct name for the means of transport, *just precise enough in local context*. In Berlin, `S-Bahn commuter rail` would be too much, because everyone knows what `S-Bahn` means. -- `short` short be the shortest possible symbol that identifies the product. +- `product`: A sensible, [camelCased](https://en.wikipedia.org/wiki/Camel_case#Variations_and_synonyms), alphanumeric identifier. Use it for the key in the `products` object as well. +- `mode`: A [valid *Friendly Public Transport Format* `1.0.1` mode](https://github.com/public-transport/friendly-public-transport-format/blob/1.0.1/spec/readme.md#modes). +- `bitmask`: HAFAS endpoints work with a [bitmask](https://en.wikipedia.org/wiki/Mask_(computing)#Arguments_to_functions) that toggles the individual products. the value should toggle the appropriate bit(s) in the bitmask (see below). +- `name`: A short, but distinct name for the means of transport, *just precise enough in local context*. In Berlin, `S-Bahn commuter rail` would be too much, because everyone knows what `S-Bahn` means. +- `short`: The shortest possible symbol that identifies the product. todo: `defaultProducts`, `allProducts`, `bitmasks`, add to profile -If you want, you can now **verify that the profile works**; I've prepared [a script](https://runkit.com/derhuerst/hafas-client-profile-example) for that. Alternatively, [submit Pull Request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) and I will help you out with testing and improvements. +If you want, you can now **verify that the profile works**; I've prepared [a script](https://runkit.com/derhuerst/hafas-client-profile-example) for that. Alternatively, [submit a Pull Request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) and I will help you out with testing and improvements. ### Finding the right values for the `bitmask` field @@ -129,14 +129,14 @@ all but ACME Commuter Rail 127 01111111 255 - 2^7 2^7 all but Foo Bar Metro 191 10111111 255 - 2^6 2^6 all but product C 223 11011111 255 - 2^5 2^5 all but product D 239 11101111 255 - 2^4 2^4 -all but product E 243 11110011 255 - 2^3 - 2^2 2^3 & 2^2 +all but product E 243 11110011 255 - 2^3 - 2^2 2^3, 2^2 all but product F 253 11111101 255 - 2^1 2^1 all but product G 254 11111110 255 - 2^0 2^0 ``` ## 4. Additional info -We consider these improvements to be *optional*: +I consider these improvements to be *optional*: - **Check if the endpoint supports the journey legs call.** - In the app, check if you can query details for the status of a single journey leg. It should load realtime delays and the current progress. @@ -156,14 +156,14 @@ As far as I know, there are three different types of authentication used among H ### unprotected endpoints -You can just query these if you send a formally correct request. +You can just query these, as long as you send a formally correct request. ### endpoints using the `checksum` query parameter -`checksum` is a [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code): `hafas-client` will compute it by [hashing](https://en.wikipedia.org/wiki/Hash_function) the request body and a *salt* (which means secret). **This secret can be read from the config file inside the app bundle.** There is no guide for this yet, so please [open an issue](https://github.com/derhuerst/hafas-client/issues/new) instead. +`checksum` is a [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code): `hafas-client` will compute it by [hashing](https://en.wikipedia.org/wiki/Hash_function) the request body and a secret *salt*. **This secret can be read from the config file inside the app bundle.** There is no guide for this yet, so please [open an issue](https://github.com/derhuerst/hafas-client/issues/new) instead. ### endpoints using the `mic` & `mac` query parameters `mic` is a [message integrity code](https://en.wikipedia.org/wiki/Message_authentication_code), the [hash](https://en.wikipedia.org/wiki/Hash_function) of the request body. -`mac` is a [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code), the hash of `mic` and a *salt* (which means secret). **This secret can be read from the config file inside the app bundle.** There is no guide for this yet, so please [open an issue](https://github.com/derhuerst/hafas-client/issues/new) instead. +`mac` is a [message authentication code](https://en.wikipedia.org/wiki/Message_authentication_code), the hash of `mic` and a secret *salt*. **This secret can be read from the config file inside the app bundle.** There is no guide for this yet, so please [open an issue](https://github.com/derhuerst/hafas-client/issues/new) instead. From f731b020e6a58c13cfa11ac0447645f15943f95e Mon Sep 17 00:00:00 2001 From: Jannis R Date: Sat, 17 Mar 2018 17:55:48 +0100 Subject: [PATCH 11/13] writing a profile guide: minor improvements --- docs/writing-a-profile.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index 59b5a11c..cf4dc0fe 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -1,6 +1,6 @@ # Writing a profile -**Per endpoint, there is an endpoint-specific customisation called *profile*** which may for example do the following: +**Per endpoint, `hafas-client` has an endpoint-specific customisation called *profile*** which may for example do the following: - handle the additional requirements of the endpoint (e.g. authentication), - extract additional information from the data provided by the endpoint, @@ -123,15 +123,12 @@ If you want, you can now **verify that the profile works**; I've prepared [a scr As shown in [the video](https://stuff.jannisr.de/how-to-record-hafas-requests.mp4), search for a journey and toggle off one product at a time, recording the requests. After extracting the products bitmask ([example](https://gist.github.com/derhuerst/193ef489f8aa50c2343f8bf1f2a22069#file-via-http-L34)) you will end up with values looking like these: ``` -toggles value binary notation subtraction bit(s) -all products 255 11111111 255 - 0 -all but ACME Commuter Rail 127 01111111 255 - 2^7 2^7 -all but Foo Bar Metro 191 10111111 255 - 2^6 2^6 -all but product C 223 11011111 255 - 2^5 2^5 -all but product D 239 11101111 255 - 2^4 2^4 -all but product E 243 11110011 255 - 2^3 - 2^2 2^3, 2^2 -all but product F 253 11111101 255 - 2^1 2^1 -all but product G 254 11111110 255 - 2^0 2^0 +toggles value binary subtraction bit(s) +all products 31 11111 31 - 0 +all but ACME Commuter Rail 15 01111 31 - 2^4 2^4 +all but Foo Bar Metro 23 10111 31 - 2^3 2^3 +all but product E 30 11001 31 - 2^2 - 2^1 2^2, 2^1 +all but product F 253 11110 31 - 2^1 2^0 ``` ## 4. Additional info From dbd1a2df367faf3ba8c3d1c77480fd5913cf210a Mon Sep 17 00:00:00 2001 From: Jannis R Date: Sat, 17 Mar 2018 20:56:11 +0100 Subject: [PATCH 12/13] writing a profile guide: I -> we, note about language of product names --- docs/writing-a-profile.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index cf4dc0fe..a1187a91 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -8,7 +8,7 @@ This guide is about writing such a profile. If you just want to use an already supported endpoint, refer to the [API documentation](readme.md) instead. -*Note*: **If you get stuck, ask for help by [creating an issue](https://github.com/derhuerst/hafas-client/issues/new)!** I want to help people expand the scope of this library. +*Note*: **If you get stuck, ask for help by [creating an issue](https://github.com/derhuerst/hafas-client/issues/new)!** We want to help people expand the scope of this library. ## 0. How do the profiles work? @@ -31,8 +31,6 @@ const myProfile = { } ``` -If you pass this profile into `hafas-client`, the `parseLine` method will override [the default one](../parse/line.js). - Assuming the endpoint returns all lines names prefixed with `foo `, We can strip them like this: ```js @@ -54,9 +52,11 @@ const createParseLineWithoutFoo = (profile, operators) => { profile.parseLine = createParseLineWithoutFoo ``` +If you pass this profile into `hafas-client`, the `parseLine` method will override [the default one](../parse/line.js). + ## 1. Setup -*Note*: There are many ways to find the required values. This way is rather easy and has worked for most of the apps that I've looked at so far. +*Note*: There are many ways to find the required values. This way is rather easy and has worked for most of the apps that we've looked at so far. 1. **Get an iOS or Android device and download the "official" app** for the public transport provider that you want to build a profile for. 2. **Configure a [man-in-the-middle HTTP proxy](https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/)** like [mitmproxy](https://mitmproxy.org). @@ -111,12 +111,12 @@ Let's break this down: - `product`: A sensible, [camelCased](https://en.wikipedia.org/wiki/Camel_case#Variations_and_synonyms), alphanumeric identifier. Use it for the key in the `products` object as well. - `mode`: A [valid *Friendly Public Transport Format* `1.0.1` mode](https://github.com/public-transport/friendly-public-transport-format/blob/1.0.1/spec/readme.md#modes). - `bitmask`: HAFAS endpoints work with a [bitmask](https://en.wikipedia.org/wiki/Mask_(computing)#Arguments_to_functions) that toggles the individual products. the value should toggle the appropriate bit(s) in the bitmask (see below). -- `name`: A short, but distinct name for the means of transport, *just precise enough in local context*. In Berlin, `S-Bahn commuter rail` would be too much, because everyone knows what `S-Bahn` means. +- `name`: A short, but distinct name for the means of transport, *just precise enough in local context*, and in the local language. In Berlin, `S-Bahn commuter rail` would be too much, because everyone knows what `S-Bahn` means. - `short`: The shortest possible symbol that identifies the product. todo: `defaultProducts`, `allProducts`, `bitmasks`, add to profile -If you want, you can now **verify that the profile works**; I've prepared [a script](https://runkit.com/derhuerst/hafas-client-profile-example) for that. Alternatively, [submit a Pull Request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) and I will help you out with testing and improvements. +If you want, you can now **verify that the profile works**; We've prepared [a script](https://runkit.com/derhuerst/hafas-client-profile-example) for that. Alternatively, [submit a Pull Request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) and we will help you out with testing and improvements. ### Finding the right values for the `bitmask` field @@ -133,7 +133,7 @@ all but product F 253 11110 31 - 2^1 2^0 ## 4. Additional info -I consider these improvements to be *optional*: +We consider these improvements to be *optional*: - **Check if the endpoint supports the journey legs call.** - In the app, check if you can query details for the status of a single journey leg. It should load realtime delays and the current progress. @@ -149,7 +149,7 @@ I consider these improvements to be *optional*: ## Appendix A: `checksum`, `mic`, `mac` -As far as I know, there are three different types of authentication used among HAFAS deployments. +As far as we know, there are three different types of authentication used among HAFAS deployments. ### unprotected endpoints From 35a70addeee7f5051056c6ce2f694e241c6e29e6 Mon Sep 17 00:00:00 2001 From: Jannis R Date: Sat, 17 Mar 2018 21:01:23 +0100 Subject: [PATCH 13/13] writing a profile guide: S-Bahn example in German --- docs/writing-a-profile.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/writing-a-profile.md b/docs/writing-a-profile.md index a1187a91..1aca3f2e 100644 --- a/docs/writing-a-profile.md +++ b/docs/writing-a-profile.md @@ -111,7 +111,7 @@ Let's break this down: - `product`: A sensible, [camelCased](https://en.wikipedia.org/wiki/Camel_case#Variations_and_synonyms), alphanumeric identifier. Use it for the key in the `products` object as well. - `mode`: A [valid *Friendly Public Transport Format* `1.0.1` mode](https://github.com/public-transport/friendly-public-transport-format/blob/1.0.1/spec/readme.md#modes). - `bitmask`: HAFAS endpoints work with a [bitmask](https://en.wikipedia.org/wiki/Mask_(computing)#Arguments_to_functions) that toggles the individual products. the value should toggle the appropriate bit(s) in the bitmask (see below). -- `name`: A short, but distinct name for the means of transport, *just precise enough in local context*, and in the local language. In Berlin, `S-Bahn commuter rail` would be too much, because everyone knows what `S-Bahn` means. +- `name`: A short, but distinct name for the means of transport, *just precise enough in local context*, and in the local language. In Berlin, `S-Bahn-Schnellzug` would be too much, because everyone knows what `S-Bahn` means. - `short`: The shortest possible symbol that identifies the product. todo: `defaultProducts`, `allProducts`, `bitmasks`, add to profile