Topic 2 – Common Integrations in With Application Architecture

As an IT professional for over 10 years, I am presented with various solutions and complexities related to integrating applications. It always starts with identifying the manual process. Typically, an employee has to manually copy data or transfer a file from one application to another. This scenario creates a lot of risk for the employee and the organization. The employee must remember to do the manual steps and execute accurately in order to prevent loss of data or the wrong data gets imported into the system. From an organizational standpoint, this can cause downtime in services, unhappy customers, data breaches, and in some cases money being transferred to the wrong individual.

The solution is always to replace the manual steps with an automated integration. The challenge with this solution is some legacy systems legally have to remain operational or it may be too costly to replace and don’t always have newer integration capabilities.  For example, a legacy system only has capabilities to export a comma-separated values (CSV) document to a folder on a server. A newer system may only have web services capabilities through SOAP or REST, which I’ll discuss more about below. This requires a middleware software to transform the CSV document into a REST message in order to send data from the legacy system to a newer application. Even though there are more costs up front, it saves an organization time and money eliminating manual steps.

Below is a list of integration technologies that can be used to eliminate manual steps and integrate various applications.

• Flat file with SFTP/FTP – A flat file is typically used in the term of a comma separate values (CSV) or XML document. This file is transferred to an SFTP or FTP location in order for another system to pick up the file. When dealing with legacy systems, this solution is often used.
• TCP connection – This type of integration I find not as common but still exists. I personally used it for credit card processing for a tuxedo rental company. The concept is as follows the sender application will establish a TCP connection on a certain port on another server. The recipient application will listen on that server for a connection on the given port. Once the connection is established, it will load/buffer the packets of data until the connection is closed. Then it will translate the data into meaningful information in order to be used in the application itself.
• HTTP methods – This is another uncommon type of integration but sometimes can be found in the wild. HTTP methods tells a resource what type of action should take place. GET and POST methods are typically used for this type of integration. GET tells a resource to retrieve certain information based on parameters that are passed in the URL. POST tells a resource to store data that is passed as parameters in the URL. Both of these methods in my opinion were the foundation for SOAP and REST web services.
• SOAP – SOAP is a form of web service that uses XML and the concept of schema. When a resource wants to send data to another resource, it must format the XML so that it matches the recipient’s schema. The schema is a blueprint defining the type of data and the format it should be structured in order to be considered safe to consume the XML. Based on my experience, this web service is no longer considered the standard. Many organizations and applications are moving towards REST web services because of how extensible and easier to trouble shoot it can be.
• REST – REST uses HTTP to send XML, HTML, or JSON to send data. Due to the flexibility and easier maintenance, it has become the norm in web services. Much like HTTP methods, typically the REST message has the action within the text that is being sent. This becomes quite powerful since custom actions can be defined. Much like SOAP, as long as the right tags are used, there is really no limit on the amount of data that can be sent at one time.
• EDI – EDI is another form of integration mostly used in an organization’s procurement and inventory management systems. It is considered an older technology and has been around for 30 plus years. The scope of EDI is beyond what I would like to discuss with this post but I can attempt to explain it with a few sentences. Previously we discussed SOAP. SOAP has an XML schema that restrict how XML is sent between two systems. EDI is very similar with standard structures however due to the standards, they are defined by transaction type. For example, if a manufacturer wants to order more raw materials from a vendor. There is a specific standard for this transaction. Same goes for submitting a purchase order to a vendor.
• cXML – cXML is another form of integration that is similar to SOAP. There is a set of XML schemas that are used as templates. The only difference I’ve noticed is that is has an additional XML tags around the XML message. It’s not something I’ve implemented but was approached with solution when integrated procurement systems. Unfortunately, the system I was using to send the XML did not support cXML.
• Direct Database Transactions – Definitely not the most ideal form of integration solutions but direct connections to databases is another way to integrate. This requires a database server to be exposed to the internet and JDBC or ODBC are typically used to run Insert, Update, or Delete queries into the database. Without full control over an application’s ability to prevent SQL injection, this solution is usually used inside a company’s local network where both the sender and receiver applications can be customized.

There’s definitely more integrations out there but I wanted to highlight all the types of integrations I’ve used in the past. Application architects have a difficult job with understanding an application’s capabilities and determining the correct solution for integrations. When the correct integration choice is made, automation can occur and time and money can be saved by the organization.