Rapid B2B integrations with Ballerina and Choreo

Many business processes involve communicating with external parties such as suppliers, logistics providers, insurance providers, and government institutions. Unlike internal communications, these inter-organizational communications usually have to follow various regulations and standards, and could introduce delays if one party doesn’t comply. Therefore, effective business-to-business (B2B) integrations are a key part of most digital transformation projects.

Let’s take the fictional example of an electronic equipment manufacturer named Smartware. Smartware supplies equipment for multiple retailers and depends on multiple suppliers and logistics partners for fulfilling orders. Their order processing workflow can be depicted as follows:

WSO2

When a retailer sends a purchase order, Smartware’s order processing workflow is triggered. It first gets the retailer’s details from the customer relationship management (CRM) system. Then it checks the inventory levels of ordered products from the enterprise resource planning (ERP) system. If the available inventory is insufficient to fulfill the order, the list of eligible suppliers is fetched from the supplier database, and the procurement process is triggered. Once all products are available, the order is shipped via a suitable logistics provider.

If we want to automate the above order processing workflow, then all internal systems (CRM, supplier database, ERP) and external systems (retailers, suppliers, logistics providers) must be properly integrated with the workflow. If, for example, suppliers are not integrated, then the process will be slowed by manual steps. Smartware staff will need to contact suppliers manually (e.g., via email) and ask for quotations. Then the  suppliers’ staff will need to search through emails, gather details, and reply. Once quotations are received, Smartware staff will have to analyze each quotation, manually select a supplier, and place an order.

Thus a single manual B2B interaction will increase response time unpredictably and introduce errors into the whole process, even if all the other steps in the process are automated.

Challenges in B2B integrations

Although B2B integrations are a critical factor for successful digital transformations, integrations among business partners are not always straightforward. Electronic Data Interchange (EDI) is the prominent B2B message format. Multiple EDI standards, such as ANSI X12 and EDIFACT, have been introduced to standardize EDI message structures used for different business messages.

For example, X12 850 and EDIFACT ORDERS specifications provide message structures to be used in purchase order messages. Despite these standardization efforts, organizations use variations of these standard message formats depending on organizational and regional regulations. Therefore, when two business partners send purchase orders using the EDIFACT ORDERS message, the actual message structure can vary, forcing the receiving organization to deploy separate processing logic per partner.

Testing B2B integrations also can be challenging due to the involvement of multiple organizations. When partner organizations use variations of EDI standards, these variations and related assumptions aren’t always captured in the partner-specific specification. Multiple rounds of schema modifications and testing may be required before deploying B2B integrations into production.

In addition, business partners may use various security mechanisms and protocols in B2B communications. For example, some partners may use SFTP, while others may use Amazon S3 locations to exchange business messages. There could be partners that use mutual HTTPS or AS2. On top of all these challenges, some organizations may not use EDI and may rely on custom JSON, XML, or CSV formats. Therefore, in addition to separate processing logic, many business partner integrations could require separate modules with their own security and transport protocols.

All of these challenges make onboarding new business partners and integrating them securely with internal IT systems a complex and time-consuming task. However, from a business perspective, it is crucial to onboard new business partners as soon as possible in order to automate end-to-end business transactions.

The remaining sections of this article describe an architecture and a methodology for quickly onboarding business partners into an organization’s internal IT platform. Two main technologies are used in this architecture: the Ballerina integration language and the Choreo internal developer platform. The Ballerina language can be used to quickly develop partner-specific EDI processing modules, while Choreo can be used to rapidly test and deploy EDI modules into production.

Creating partner-specific B2B EDI packages

Ballerina is a programming language specifically designed for writing integrations. It provides built-in support for network data types such as JSON, XML, CSV, and EDI. In addition, Ballerina provides powerful data handling capabilities with flexible record types and transformations, as well as a large collection of protocol and system connectors. It also provides tool support and code generation related to various network protocols and data types such as OpenAPI, GraphQL, gRPC, and EDI. Therefore, the use of Ballerina can greatly simplify B2B integrations.

The steps for creating a partner-specific Ballerina EDI package are shown below.

WSO2

Whenever you onboard a new business partner, it is necessary to get the EDI specifications used by that partner. In most cases, companies use variations of standard EDI documents such as EDIFACT and X12. EDI schemas for these common EDI messages are readily available in Ballerina. If the new business partner uses variations of these standard messages, the schemas available in Ballerina can be changed to support those variations. Once the required EDI schemas are prepared, the Ballerina EDI tool can be used to generate a Ballerina project that can translate EDI messages into JSON and vice versa.

If it’s feasible to work directly with the JSON equivalent of EDI messages, the generated EDI project can be built into a Ballerina package, deployed, and used without further modifications. However, in most cases, it is necessary to map EDI messages to internal message formats before processing further.

For example, the EDIFACT INVOIC message used by a supplier can have a very large number of segments and fields. Furthermore, some of these fields may not be compliant with the standard and could be specific to the supplier. Usually it is necessary to transform EDI data into completely different data structures, which are used internally within the receiving organization. This approach also facilitates the onboarding of partners with multiple variations of EDI messages, as each partner’s EDI messages can be transformed into partner-independent internal formats. The Ballerina data mapping tool provides a visual and intuitive way to map such partner-specific EDI messages into any target message structures, as shown in the figure below.

WSO2

The final part of preparing the EDI package is to include transport mechanisms used by the business partner. HTTP is supported by default by the generated package. If a partner uses HTTP, then the generated project can be used as is. However, if a partner needs to use FTP, AMQP, Amazon S3, or any other protocol, that protocol support must be added to the generated package. Because Ballerina has built-in support for many protocols and connectors, supporting additional protocols in the EDI package usually takes only a few lines of code.

Rapid testing and deployment of B2B EDI packages

Once the above steps are completed, we get a Ballerina project for an EDI processing package that can process partner-specific EDI messages and convert those messages to internal messages used within the organization. Now it’s time to test this EDI processing module and deploy it into production once everything is working as expected.

Choreo can be used as the B2B developer platform that provides built-in CI/CD, security, developer onboarding, and support for multiple environments. Choreo can work directly with GitHub repositories, allowing you to create a Choreo project from the GitHub location of a Ballerina B2B EDI project. The deployment of B2B projects in Choreo is illustrated in the figure below.

WSO2

First, Choreo’s CI/CD pipeline builds the Ballerina EDI project given in the GitHub location and deploys it into the development environment. At this point, the B2B integration will be executed using the development environment parameters (e.g., database connection details and FTP location), which can be configured in Choreo. Once development testing is complete, the project can be promoted to a higher-level Choreo environment for additional testing.

For example, this could be an environment for testing with business partners, which points to FTP locations within partner organizations. If an issue is discovered during partner testing, you can make changes in any of the previous steps (e.g., EDI schema modification) and propagate those to the partner testing environment via the Choreo CI/CD pipeline.

Finally, when partner testing is completed, the Choreo pipeline can promote the B2B project to the production environment, which points to production databases and production EDI source and target locations. Here you can use Choreo’s observability features to monitor logs and message statistics, crucial for B2B integrations.

In addition to B2B modules, any other component developed in any language or technology can be deployed in the Choreo platform and managed alongside B2B modules. For example, the Choreo platform diagram above shows an invoice processing component deployed alongside B2B modules. Different variations of EDIFACT invoice messages sent by suppliers can be converted to a common internal message format by corresponding EDI modules. Then these internal invoice messages are sent to the invoice processing component, which retrieves the relevant fields from the invoice message and stores them in a database.

There are many tools, libraries, and products for building B2B integrations. Criteria for selecting a particular technology for a B2B project must include the flexibility of incorporating partner-specific message formats, ease of data transformations, support for multiple protocols, and most importantly, the ability to onboard partners quickly by working collaboratively with external parties. Ballerina and Choreo satisfy all of those requirements and provide a comprehensive environment for developing B2B integrations.

Chathura Ekanayake is an associate director/architect at WSO2. He is a part of the BPS team and focuses on overall BPM related aspects. He first joined WSO2 in 2006 and led the development of the WSO2 governance registry product in addition to working on WSO2 ESB. He pursued a Ph.D. at the Queensland University of Technology, Australia, and joined WSO2 upon completion of the Ph.D. in 2013. Chathura holds a first class honors degree in computer science and engineering from the University of Moratuwa, Sri Lanka.

—

New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.