D365F&O. File based integration using Azure Blob Storage

Disclaimer

This article would be interesting for technical specialists. It does not contain exact example of end-to-end integration. Article has only code example with comments of getting file from user by implementing core classes for work with Azure Blob Storage.

Requirements

You have to be familiar with X++ language and have basic understanding of .NET Framework.

Problem

Dynamics 365 for Finance and Operations (and all another 'versions' driven by marketing department) has a very powerful out of box integration framework. It is called Data Management Framework. My suggestion is to use it for your file based integrations.

The common task nowadays is upgrade from AX 2012 (on-prem) to D365F&O (cloud). The recurrent file-based integrations (X++, DMF, or AIF) can be converted to Data Management Framework. However, there is a one type of integration, which requires more effort. It is a file based integration via middle layer FTP server.

The general idea of such integration is about placing file to the predefined folder on the FTP server. For example, we have an internet shop which is driven by its own engine. This shop generates a Sales Order. The internet shop generates the file which contains info about sales order and places it to the specific folder of the FTP server. AX 2012 picks up the file and creates SO.

Yes, it is still possible to implement same scenario for the D365F&O. This is an advantage of such scenario. We will not touch external system (internet shop) at all. However, this scenario contains several major disadvantages:

1.     We are moving ERP to the cloud, but still have to manage a FTP server

2.     We have to use third party .library for allowing D365F&O connection to FTP

3.     Are you kidding? 2021 and you are still using FTP

The alternative is using an external Azure Blob Storage (external because D365F&O uses internal storage for document attachments). The disadvantage is only one (at least, for projects where I was involved) we have to change external system to use Azure Blob Storage. However, the external system (in my cases) is able to work with Azure Blob Storage out of box. All we needed was a correct setup.

The advantages of using Azure blob storage next:

1.     The subscription is lower than price of maintaining FTP (for my cases)

2.     We do not have to use third party library

3.     Modern cloud-hosted storage system

4.     It is possible to use business events for more intelligent integration

5.     A high chance that external system already can work with Azure Blob Storage without an impact

As you can understand, we decided to move integration from FTP to Azure Blob Storage

Solution

Before moving to the coding, let me define basic terms of the Azure Blob Storage. This is simplified version of original definition, but it allows having constant picture of the integration.

Connection string is our path to the Azure Blob Storage. It should contain address and access token (or user name and password. Unfortunately, I cannot remember it now, and, as usual, I cannot find the backup of the setup table for it)

Container is our folder of the Azure Blob Storage. It is not a folder. It is container. However, it is much more simply imagine it like an old fashion folder.

Blob Block is our file. However, it is not the file itself. It is a metadata of the file and the refernce to it.

Now we are ready for implementing our classes.

The first is AzureBlobStorageHelper. It is a core class for all operations, like establishing connection, uploading and downloading file, moving file between containers, deleting file.

Let us start from the class definition

using Microsoft.WindowsAzure.Storage;

/// <summary>

/// A helper class for Azure BLOB storage actions

/// </summary>

class AzureBLOBStorageHelper implements System.IDisposable

{

   protected Microsoft.WindowsAzure.Storage.CloudStorageAccount storageAccount;

   protected str connectionString;

   protected boolean silentMode;

}

As you can see, we have to use a Microsoft.WindowsAzure.Storage namespace. You can argue that we are connecting the third party library. But it is not true. We are using an out of box library which is provided by Microsoft. That allows us be confident during deployment that everything would be in place, because Microsoft.WindowsAzure.Storage namespace available on vanilla version.

The class itself has three states

·        storageAccount. It is our Azure Blob Storage engine. It will execute all required operations without any effort from our side. It should not have a corresponding parm-method

·        connectionString. It is our Connection String. It has corresponding parm-method

·        silentMode. The requirement was not showing some messages during processing. As well as logging errors to the log table. However, I forgot to get the copy with logging mechanism, so example will contain only hiding info messages. It has corresponding parm-method

Note: it is not mandatory to extend from IDisposable interface. I did just for the possibility of calling dispose method for forcing closing connection.

Now we can implement construct method

/// <summary>

/// Creaets an instance of <c>AzureBLOBStorageHelperNGT</c> and initializes it

/// </summary>

/// <param name = "_connectionString">An Azure BLOB storage connection string [optional]</param>

/// <param name = "_silentMode">Determines messages (info/warning) should not be shown (true) or should be (false)[optional]</param>

/// <returns>An instance of <c>AzureBLOBStorageHelperNGT</c></returns>

public static AzureBLOBStorageHelper construct(str _connectionString = '', boolean _silentMode = false)

{

           AzureBLOBStorageHelper storageHelper = new AzureBLOBStorageHelper();

           

           if (_connectionString != '')

           {

                       storageHelper.connectionString = _connectionString;

           }

           storageHelper.silentMode = _silentMode;

           storageHelper.init();

           return storageHelper;

}

Both parameters of the construct method are optional.

The connection string is optional because we have to provide the possibility of getting default connection string. By default we are showing all messages to the user.

The key method of the constructor is init method. Is method initializes storageAccount state.

/// <summary>

/// Initializes instance of the class

/// </summary>

protected void init()

{

           if (!connectionString)

           {

                       connectionString = AzureBLOBStorageHelper::getDefaultConnectionString();

                       if (!connectionString)

                      {

                                 throw error("Azure BLOB stroage connection string is not specified");

                      }

           }

           storageAccount = CloudStorageAccount::Parse(connectionString);

           if (!storageAccount)

           {

                       throw error("Cannot establish connection to Azure BLOB storage");

           }

}

Now we are ready to download file from the storage.

/// <summary>

/// Downloads file (as a stream) from Azure BLOB storage

/// </summary>

/// <param name = "_fileName">A file name</param>

/// <param name = "_containerName">A container name</param>

/// <returns>A file stream</returns>

public System.IO.Stream downloadFile(Filename _fileName, str _containerName)

{

           if (!storageAccount)

           {

                       throw error("System does not have active Azure BLOB storage connection");

           }

           if (!_containerName)

           {

                       throw error(Error::missingMethodParameter(classStr(AzureBLOBStorageHelper), methodStr(AzureBLOBStorageHelper, downloadFile), 'ContainerName'));

           }

           // Create a reference to the file container

           Blob.CloudBlobClient   blobClient     = storageAccount.CreateCloudBlobClient();

           Blob.CloudBlobContainer blobContainer  = blobClient.GetContainerReference(_containerName);

           if (!blobContainer.Exists(null, null))

           {

                       throw error(strFmt("Container %1 is not found.", _containerName));

           }

           // Create a reference to the file client.

           Blob.CloudBlockBlob blobBlock = blobContainer.GetBlockBlobReference(_fileName);

           if (!blobBlock

            || !blobBlock.Exists(null, null))

           {

                       throw error(strFmt("File %1 is not found.", _fileName));

           }

           System.IO.Stream fileStream = new System.IO.MemoryStream();

           blobBlock.DownloadToStreamAsync(fileStream).Wait();

           if (!silentMode)

           {

                       blobBlock.FetchAttributes(null, null, null);

                       Blob.BlobProperties blobBlockProperties = blobBlock.Properties;

                       if (blobBlockProperties.Length == fileStream.Length)

                       {

                                  info(strFmt("File %1 is downloaded", _fileName));

                       }

           }

           return fileStream;

}

This is where magic happens. First of all, we have to check do we still have connection to the storage and does caller defines source container. The next step is opening the blob storage client. The client will execute all necessary actions based on inputs (connection, container, file name).

Because I would like to be 100% sure that everything is ok, I am checking if container exists before getting blob block. Again, before returning file itself (the FileStream with data) we are checking if we got blob block and if blob block exists.

The final step (before returning FileStream) is a downloading file. The blob block object supports async downloading. However, it is not good idea to return stream before downloading finished. Because of that we have to call Wait().

The class supports all basic actions. You can download the whole code from link at the end of topic.

However, integration which I had to upgrade was a partially manual. E.g. I had to show all files from the specific container and only after user would choose his destiny (oh sorry) the file to import.

Because of that we have to prepare the list of available files. The information about each file should contain file name and creation date and time. Because of that let us prepare a class-container. This class will store the info about file.

/// <summary>

/// An file info class for Azure BLOB storage objects converting to X++

/// </summary>

class AzureBLOBStorageFileInfo

{

   public Filename               fileName;

   public System.DateTime dateTime;

   public str                          fileUri;

}

This class contains three states

·        fileName for file name

·        dateTime for creation date time (upload date time)

·        fileUri for saving whole URL path to the file (I cannot remember where did we use it)

All we need is a function which will return us a list of AzureBLOBStorageFileInfo which comes from same container

/// <summary>

/// Gets list of files (file infos) for specific container

/// </summary>

/// <param name = "_container">A container</param>

/// <returns>A List of <c>AzureBLOBStorageFileInfoNGT</c></returns>

public List getFileInfoList(str _container)

{

           List ret = new List(Types::Class);

           if (!storageAccount)

           {

                       throw error("System does not have active Azure BLOB storage connection");

           }

           if (!_containerName)

           {

                       throw error(Error::missingMethodParameter(classStr(AzureBLOBStorageHelper), methodStr(AzureBLOBStorageHelper, getFileInfoList), 'ContainerName'));

           }

           Blob.CloudBlobClient   blobClient     = storageAccount.CreateCloudBlobClient();

           Blob.CloudBlobContainer blobContainer  = blobClient.GetContainerReference(_container);

           if (!blobContainer.Exists(null, null))

           {

                       throw error(strFmt("Container %1 is not found.", _container));

           }

           System.Collections.IEnumerable list = new System.Collections.Generic.List<str>();

           list = blobContainer.ListBlobs(null, false, 0, null, null);

           System.Collections.IEnumerator listEnumerator = list.getEnumerator();

           while(listEnumerator.moveNext())

           {

                      Blob.CloudBlockBlob blobBlock = listEnumerator.get_Current();

                       if (blobBlock.Exists(null, null))

                       {

                                  Blob.BlobProperties blobBlockProperty = blobBlock.Properties;

                                  AzureBLOBStorageFileInfo fileInfo = AzureBLOBStorageFileInfo::construct();

                                  fileInfo.fileName  = blobBlock.Name;

                                  fileInfo.fileUri   = blobBlock.StorageUri.PrimaryUri.AbsoluteUri;

                                  fileInfo.dateTime  = blobBlockProperty.LastModified.HasValue ? blobBlockProperty.LastModified.Value.DateTime : nullValueBaseType(Types::UtcDateTime);

                                  ret.addEnd(fileInfo);

                       }

           }

           return ret;

}

The logic is pretty same to downloading file except the fact, that we are asking blob storage to give us an info about files by method ListBlobs. Based on the result we are preparing the list of X++ objects instead.