# Operations for data processing The whole point of using Redox is to process your data so you and your connection can speak to each other in the respective data formats you can support. More than that, you each have different needs for what data looks like or does before it gets to you. To do that, Redox relies on data operations. ## What is a data operation? A **data operation** is an individual step during log processing. There are multiple types of operations, but they all have the same goal of shaping data appropriately so you can successfully send and receive data. Typically, multiple operations are applied to an individual request. But not every log goes through the same set of operations. The operations applied to a log depend on the subscription’s settings. ## Who owns data operations Redox manages most operations for you. You can still see them in **log inspector** so you can pin down where or why data changed unexpectedly for troubleshooting purposes. [Learn about log inspector](https://docs.redoxengine.com/permalink/790exE7anSFdJdupyjnidt/#what-is-log-inspector). However, you know your data needs best, so you can create and manage some of these operations yourself: - config modifiers - filters - translation sets Review the categories and operation types below for more details about each. ## Mapping and modifying operations Mapping and modifying operations define how to convert and format data appropriately, as well as perform field mapping. ### Base configs _Managed by_: **Redox** For each data exchange, Redox normalizes data for a consistent experience—it’s the Redox promise! A **Redox base config** operation includes converting a payload from a given system’s format to the appropriate Redox data model or FHIR resource. These conversions happen on both sides of the exchange so that you or your connection gets data in the expected format. A Redoxer can link a relevant base config to your non-API sources and destinations to ensure proper message processing.
Available base config options You need a base config link for each type of HL7v2 message you’re normalizing to FHIR from this source. Base configs are named for the type of HL7v2 message you’re sending and the FHIR notification to normalize your data to ([see our FHIR notifications](https://docs.redoxengine.com/permalink/fhir-notification-landing-page)). Choose one of these available base configs for sources: - MDM to FHIR - ORM RDE to FHIR - ORM to FHIR - RDE to FHIR - ADT to FHIR - ORU to FHIR - SIU to FHIR - DFT to FHIR Just remember you’ll need a subscription for every base config link you add. [Learn how to create a subscription](/how-to-use-redox/manage-your-redox-setup/create-a-subscription).
### Config modifiers _Managed by_: **You or your connection** A **config modifier** is a set of custom instructions you create for processing incoming or outgoing data. This operation is typically applied on top of a Redox base config, but it may be applied independently in some cases. Depending on the config modifier flavor, it can either write new data to or delete existing data from a payload. [Learn about config modifiers](/how-to-use-redox/change-data-with-config-modifiers). > **Confidential data in config modifiers** > > The contents of a config modifier might contain confidential data, like personal health information (PHI). Because of that, the contents are only visible to the organization that manages the config modifier. > > If your organization doesn’t own the config modifier, you’ll see its name, flavor, and status without the contents in the operation details of log inspector. > **Canadian PHI** > > We store data for customer-managed operations (e.g., translation sets, filters, config modifiers) in the U.S. with the same safeguards as PHI data. Given that Redox configuration data is housed in the U.S., though, it might not be appropriate to include Canadian patients’ PHI in any of these settings. Please consult with your legal counsel before inputting any PHI originating in Canada. ### Translations _Managed by_: **You or your connection** Translation sets allow you to translate one system’s preferred value set to another system’s value set. Furthermore, a translation set defines how those values should map to each other. [Learn about translation sets](/how-to-use-redox/create-translation-sets-and-links). A **translation** operation occurs when a translation set you’ve created applies to a log payload. Translations contain explicit logic to change a value that’s always the same. They aren’t conditional and don’t alter the shape of data like config modifiers. Translations can happen on either side of the data exchange, as well as for outgoing or incoming values. > **Confidential data in translations** > > The contents of a translation set might contain confidential data, like personal health information (PHI). Because of that, the contents will only be visible to the organization that manages the translation set. > > If your organization doesn’t own the translation set, you’ll only see its name and metadata in the operation details of log inspector. > **Canadian PHI** > > We store data for customer-managed operations (e.g., translation sets, filters, config modifiers) in the U.S. with the same safeguards as PHI data. Given that Redox configuration data is housed in the U.S., though, it might not be appropriate to include Canadian patients’ PHI in any of these settings. Please consult with your legal counsel before inputting any PHI originating in Canada. ## Cleaning and structuring These operations polish incoming or outgoing requests so there aren’t any snags during delivery. ### Cleansers _Managed by_: **Redox** One system’s fundamental data is another’s extraneous data. Some payloads need pruning or sprucing to fit our Redox data model or FHIR resource format during transit. A **cleanser** operation both removes unknown fields from payloads and creates empty placeholders for missing fields. It also sets `null` for empty or missing fields in many cases. Sometimes errors are returned if validation fails significantly during a cleanser operation. ### JSON converters _Managed by_: **Redox** The **JSON converter** operation generates a JSON representation of data from a given inbound data format or to a different outbound format. The operation details will show whether it’s applied as **from JSON** or **to JSON**. For example, we convert HL7 data in pipe-delimited format to JSON like this: **From HL7v2** ```yaml MSH|^~\&|EETS|SENDER-FACILITY|||20240314150301||ADT^A08 ``` **To JSON** ```yaml { "MSH": { "1": "|", "2": "^~\&", "3": "EETS", "4": "SENDER-FACILITY", "7": "20240314150301", "9": "ADT^A08" } } ``` Conversely, running from JSON mode converts HL7 data that's structured as JSON into HL7-formatted data, like this: **From JSON** ```yaml { "MSH": { "1": "|", "2": "^~\&", "3": "EETS", "4": "SENDER-FACILITY", "7": "20240314150301", "9": "ADT^A08" } } ``` **To HL7v2** ```yaml MSH|^~\&|EETS|SENDER-FACILITY|||20240314150301||ADT^A08 ``` ### Pre-parsers _Managed by_: **Redox** Redox likes to be preemptive about catching troublesome formats or content. So, the **pre-parser** operation runs a Redox-written code snippet to clean up or adjust a payload before the JSON converter operation. These code snippets are “early fixers” that adjust payloads to increase the chances of successful processing. ### Taggers _Managed by_: **Redox** In the olden days, we used to address envelopes to let the postal office know where a letter should go. The postal office would stamp envelopes to let the receiver know when they processed the letter. Likewise, the **tagger** operation adds metadata about processing, like source and destination IDs, timestamps, and setting data model and event type names if they're missing. This doesn’t include any changes to values in payload fields. The goal of this operation is to ensure metadata consistency. ## Routing and sending Now that we’ve defined what to do and polished it up, we’re ready to route and send using these types of operations. ### BLOB resolvers _Managed by_: **Redox** Binary Large Object (BLOB) files are things like PDFs or images that don’t fit neatly into the data model or FHIR resource structure. Typically, you can upload BLOB files to our BLOB file API, then include a reference link directly in a payload to send a file. [Learn about sending a file](/basics/data-exchange-with-redox/sending-a-file). A **BLOB resolver** operation takes references to BLOB files in a payload and looks for a matching file in the BLOB API. The outcome of this operation lets you know whether Redox successfully locates a matching file. The actual file contents aren’t shown in the log inspector window. If you’re assigned to an engineer or support role, though, you can click a link to view the file’s contents. ### Conductors _Managed by_: **Redox** We all want our packages to get to the correct destination. In technical terms, Redox relies on a bit of directing between sources and destinations to make sure data gets to the correct place. A **conductor** operation finds the destination(s) that the log should go to by looking at the subscription's details. A subscription defines which source talks to which destination using a given data model or FHIR resource. You may have a subscription for the same data model going to multiple destinations. In that case, Redox sends a copy of the same log to each destination. The details of a conductor’s operation show what the subscription matches to the log. It also captures “slug routed” FHIR logs that use part of the URL to identify where the log should go. ### Deliveries _Managed by_: **Redox** A **delivery** operation is like your Amazon driver; they deliver the package to the right place at the right time. Similarly, delivering to the target destination is the end goal of a `SEND` request. Delivery is the core operation of the `RECEIVE` stage, even though other operations run during `RECEIVE` as well. Technically speaking, delivery is the actual transport of the request to the destination. This is just like your Amazon delivery driver setting your package on your porch: their delivery happens at the same time you receive the package. Delivery results in a response from the destination, whether a generic success message or an error. > **Tip about errors** > > If you see an error in the deliveries step, you might assume there was a failure during Redox processing (i.e., in building the request or translating the response). However, if you see a response with an actual reply object, it means Redox successfully delivered the request but the destination system sent back an error. > > Please know this isn’t us pointing fingers. Since knowledge is power, we want you to know where the error occurred so you can better troubleshoot. The delivery operation captures both what Redox sent and the response we got back. So this operation exists to capture those two payloads: the input payload snapshot shows what we sent to the destination, and the output payload snapshot contains the response we received back. > **Differences with log inspector** > > Log inspector is a troubleshooting tool to help you investigate when and what payload changes happened during log processing. > > With log inspector, the delivery operation is slightly different, since Redox doesn't actually send the message to the destination. The log inspector delivery details only show contents from the original log. For example, if the original log failed to deliver due to a timeout error, log inspector will always show a timeout error at that operation. > > [Learn how to run log inspector](/how-to-use-redox/run-log-inspector). ### FHIR pagination resolvers Some FHIR queries would return too many results, so FHIR relies on pagination. [Learn about FHIR pagination](https://docs.redoxengine.com/permalink/7ALppO2kO7q451QkoLuRYc/#pagination-links). A **FHIR pagination resolver** takes a FHIR server's pagination link and resolves it to a Redox-centric link instead. You can use these links to review pages of search results. ### Filters _Managed by_: **You or your connection** Filters are rules that you define to allow or block a subset of data payloads that you receive asynchronously from one of your connections. [Learn about filters](/how-to-use-redox/define-and-manage-filters). A **filter** operation occurs when a filter you’ve created runs on a log. This type of operation only happens in the `REQUEST` stage of log processing. > **Confidential data in filters** > > The contents of a filter may contain confidential data, like personal health information (PHI). Because of that, the contents will only be visible to the filter owner. If your connection owns the filter, you'll only see its name and metadata in the operation details of log inspector. > **Canadian PHI** > > We store data for customer-managed operations (e.g., translation sets, filters, config modifiers) in the U.S. with the same safeguards as PHI data. Given that Redox configuration data is housed in the U.S., though, it might not be appropriate to include Canadian patients’ PHI in any of these settings. Please consult with your legal counsel before inputting any PHI originating in Canada. ### Redox filters _Managed by_: **Redox** Like your custom filters, Redox may also define filter rules for a log. They function mostly the same, but are managed by Redox. A **Redox filter** operation occurs when a filter Redox manages runs on a log. This type of operation can happen on either the source or destination side of the exchange, whether for outgoing or incoming requests. A Redox filter can stop a log from processing further or can modify the log content itself. Just note that the contents of a Redox filter won’t be visible in the Redox dashboard.