Translate gear

The translate gear moves values from the fields of one protocol to the fields of another. A translation spec says which alias of one protocol spec becomes which alias of the other. The gear does not know any protocol: it reads aliases and writes aliases, and it imports no codec. The codecs on each side convert the format.

The same gear serves any pair of protocol specs. Examples are ISO 8583 to ISO 20022 and back, one ISO 8583 dialect to another, and one ISO 20022 message to another inside one protocol.

Attribute	Details
Type	`translate`
Status	Stable
Source Code	pkg/gears/native/translate
Pairs With	ISO8583 codec gear, ISO 20022 codec gear, Conductor gear
Always Emitted Metadata	`translate.spec_id`, `translate.spec_version`, `translate.spec_hash`, `translate.name`, and the identity of the target message (`mti`, or `iso20022.message`)
Conditionally Emitted Metadata	`iso20022.participant_id` (when the translation spec has a `participant_id` and the target is ISO 20022), `translate.violations` (when validation is on and a check failed), `error.reason`, `error.gear` and `error.message` (on the `error` port, with `on_error: reject`)
Consumed Metadata	`codec.spec_id`, `codec.spec_version` (to check the source spec)
Consumed Fields	The source aliases that the bindings name

A worked example that uses this gear is the tutorial Bridge ISO 8583 and ISO 20022.

Reference

fluxrig gears doc translate generates the identity, ports, and configuration below from the gear's manifest, so they stay in lockstep with the code.


Type	`translate`
Category	logic
Status	stable
Terminus	transparent

Translates a message between two protocol specs, alias to alias, as a translation spec says. Adds the target fields and removes none.

Ports

Port	Direction	Role	Summary
`in`	input	message	Decoded message of the source spec. Its codec.spec_id and codec.spec_version must match the spec the translation names.
`out`	output	message	The same message with the target fields added. Sets the target message identity (mti, or iso20022.message), iso20022.participant_id when the translation spec names one, and the translate.* keys. A generate binding (for example instr_id) runs here, before any conductor parks the message.
`error`	output	error	With on_error: reject, the message that the gear rejects, with error.reason (violation or translation_failed), error.gear and error.message. translate.violations lists the findings.

Configuration

Field	Type	Required	Default	Description
`spec_path`	string	yes	-	The translation spec: a path to a file, or a store reference as name:tag (e.g. billpay-rtp:v1.0.0) or a content hash. The protocol specs and the lookup sources it names resolve the same way: a file path or a store reference.
`on_error`	enum: `reject`, `drop`, `kill`		`drop`	on a translation that fails, or on a finding that rejects: reject (emit the message on the error port, with error.reason), drop (discard it), or kill (return the error to the runtime).
`translation`	string		`auto`	the name of one translation in the spec, or auto to choose the translation whose from message is the message that arrives.
`validation`	enum: `off`, `warn`, `enforce`		`off`	What the gear does when the output misses a field that the target message requires, or when another spec version decoded the message. off: no checks. warn: the gear records the findings in translate.violations, and the message continues. enforce: a finding that rejects fails the message, which then follows on_error.

The translation spec

A translation spec is a document of the fluxrig spec family with kind: translation. It has its own name, version and hash, and fluxrig stores it like a protocol spec. It refers to the protocol specs that it connects, and it changes neither of them.

spec:
  kind: "translation"
  id: "billpay-rtp"
  version: "1.0.0"
  specs:
    card: "iso8583-v87-ascii:v3.0.0"
    rail: "iso20022-rtp:v1.0.0"
  participant_id: "99000000101"
  lookups:
    billers:
      key: "card:merchant_id"
      fields: ["name", "routing", "account"]
      source: "billers:v1"
  tables:
    tx_status_to_resp_code:
      "ACTC": "00"
      "RJCT/AM02": "13"
      default: "05"
  translations:
    payment_request:
      from: "card:0200"
      to: "rail:pacs.008.001.08"
      bindings:
        - { to: "instr_id", generate: "instr_id" }
        - { to: "end_to_end_id", from: "rrn" }
        - { to: "settlement_amount", from: "txn_amount",
            format: "amount", currency: "txn_currency" }
        - { to: "creditor_name", lookup: "billers.name" }
        - { to: "debtor_name", value: "Example Payments Operator" }
    payment_status:
      from: "rail:pacs.002.001.10"
      to: "card:0210"
      bindings:
        - { to: "resp_code", from: ["tx_status", "status_reason"],
            table: "tx_status_to_resp_code" }

Key	Description
`kind`	Always `translation`. A spec with no `kind` is a protocol spec.
`id`, `version`	The name and version of the translation. The spec store keeps it under `id:v<version>`.
`specs`	The protocol specs that the translation connects, each under a short name. A reference is a store reference, `name:tag` (for example `iso8583-v87-ascii:v3.0.0`) or a content hash, or a path to a file.
`participant_id`	The participant identifier of the sender on an ISO 20022 rail. `generate: instr_id` uses it, and the gear passes it to the ISO 20022 codec in metadata. Optional.
`lookups`	Records that the source message does not carry. Each lookup has a `key` (a source alias, with the name of its spec), the `fields` of a record, and the `source` of the records: a store reference or a path to a file.
`tables`	Value tables. Each maps a value to another, with an optional `default`.
`translations`	The translations, by name. Each has a `from` message, a `to` message and its `bindings`.

Messages

from and to each name a message as <spec>:<message>, for example card:0200 or rail:pacs.008.001.08. The message must be in the catalog of its spec.

The gear sets the identity of the target message itself: the MTI for an ISO 8583 target, or the iso20022.message metadata for an ISO 20022 target. No binding sets it.

One translation spec can carry the two directions of a bridge as two translations. When the catalogs pair the messages (0200 with 0210, pacs.008 with pacs.002), the loader checks that the two translations agree with the pairing.

Bindings

A binding writes one target alias. to names an alias of the target spec. The rest of the binding says where the value comes from.

Kind	Example	Value
`from`	`{ to: "end_to_end_id", from: "rrn" }`	The value of a source alias.
`value`	`{ to: "debtor_name", value: "Example Payments Operator" }`	A constant.
`lookup`	`{ to: "creditor_name", lookup: "billers.name" }`	A field of the lookup record that the key of the lookup finds.
`generate`	`{ to: "instr_id", generate: "instr_id" }`	A generated value: `instr_id` (an instruction identifier in the format of the target profile, from `participant_id`), `now`, `today`, or `uuid`.

Two modifiers change a value:

Modifier	Example	Effect
`table`	`{ to: "resp_code", from: ["tx_status", "status_reason"], table: "tx_status_to_resp_code" }`	Translates the value through a table. With a list of source aliases, the gear joins their values with `/` before it looks them up, and it uses `default` when no entry matches.
`format: amount`	`{ to: "settlement_amount", from: "txn_amount", format: "amount", currency: "txn_currency" }`	Converts an ISO 8583 amount in minor units, with its numeric currency at `currency`, to an ISO 20022 decimal amount and its alphabetic currency. The conversion works in both directions.

A binding never names a DE number or an XML path, only aliases. The same translation thus works for every dialect whose spec declares the same aliases.

How the gear applies the bindings

All reads come first. Every binding reads the message as it arrived. The gear writes after all the reads. Thus a translation that exchanges two values gives the same result in any order of its bindings.
The gear adds and removes nothing else. It writes the target aliases and keeps every field the message had. A conductor downstream still finds its correlation alias. Only what the target codec encodes goes on the wire, and that is the fields of the target spec.

Lookups

A lookup supplies data that the source message does not carry, for example the bank account of a biller. The translation spec holds the definition of the lookup: its key and its fields. The records are a separate artefact, because they change more often than the translation. A change to the records does not change the hash of the translation.

The records are a YAML map. Each key is a value of the lookup key, and each record maps the fields of the lookup to their values:

"BILLER000000001":
  name: "Example Water Utility"
  routing: "990000002"
  account: "7700123456"

source names the records as a store reference or as a path to a file. A store reference lets the records travel to the Racks like the specs: upload them to the Mixer with a name and a tag, and the Mixer sends them with every scenario whose translation names them. See Uploading to the Mixer.

The gear loads the records when it starts, and it logs the source and the hash of the records. A record with a field that the lookup does not declare stops the start. A key that has no record leaves its bindings without a value. With validation on, the gear reports the target fields that stay empty.

What the gear checks

When the spec loads

The gear does not start when the translation spec has one of these errors. The error names the symbol that is wrong.

A binding names an alias that its spec does not declare.
A from or to message is not in the catalog of its spec.
A reference in specs or a lookup source does not resolve.
The key of a lookup is not an alias of the source spec.
Two translations contradict the pairing of the catalogs.

A field that the target message requires and that no binding writes is not an error when the spec loads. Another gear can supply it: for example, the conductor restores the parked ISO 8583 fields of an answer before the translation. The gear checks the required fields on each message instead.

On each message

With validation: warn or enforce:

The output. A field that the target message requires, and that is still empty after the translation, is a finding.
The source spec. The gear compares codec.spec_id and codec.spec_version on the message with the source spec that the translation names. A message that another version of the spec decoded is a finding.

The gear records the findings in translate.violations and logs them. With warn, the message continues. With enforce, a finding fails the message, which then follows on_error.

With on_error: reject, a message that fails leaves on the error port. error.reason is violation for a finding with enforce, and translation_failed for a translation that could not run, for example a currency that the amount conversion does not know. Wire error to a gear that answers the sender: the bridge tutorial answers a biller that is not in the directory with response code 03.

Place in a pipeline

The gear comes after the decode of the source and before the encode of the target. A conductor that matches requests and answers goes between the translation and the encode on the request path. On the answer path, it goes between the decode and the translation:

request:  decode -> translate -> conductor -> encode -> I/O
answer:   I/O -> decode -> conductor -> translate -> encode

The conductor then handles only structured messages, and a value that the translation generates, such as instr_id, exists before the conductor parks the request.

Reference​

The translation spec​

Messages​

Bindings​

How the gear applies the bindings​

Lookups​

What the gear checks​

When the spec loads​

On each message​

Place in a pipeline​