#DevOps Series: Building Dexterity Applications with Visual Studio Team Services Part 1/3

Resuming my series, I wanted to touch base on the process of building and releasing Dexterity applications with Visual Studio Team services. My friend and fellow MVP, David Musgrave explained how to setup your Dexterity development environment in his Dexterity Development series and the first article in this series directly addressed the source code control process. Although David showed some really clever methods to build and package your Dexterity chunk application, that process still has some downsides to it. Primarily, the process is dependent on a person and a physical machine dedicated to executing the process.

---

Setting up a Build Engine

When creating a self-contained cloud-based VSTS Build Engine for your Dexterity projects, there are a few considerations:

1. The actual Dexterity IDE. If you are building a Dexterity chunk, you need to at least have a copy of the Dexterity IDE, because you will want to compile your project dictionary prior to chunking, and the chunking process itself still relies on Dexterity Utilities to be able to produce a chunk file.

In addition, you will need as many IDEs as versions of Dynamics GP you are supporting with your product. David describes this well in Part 2 of his series.

2. You need clean (base) dictionaries of each Dynamics GP versions you will be supporting. This is particularly important as you will want to pull the source code from the repository into clean dictionaries to compile and obtain your extracted dictionaries and finally complete the auto-chunk process.

3. Since your build process will ultimately be automated, you need macros to inject constants and compile the dictionary, based on the version of Dynamics GP your product will be supporting. You will also need macros to extract and auto-chunk your product.

NOTE: You can inject constants into your development dictionary by having a Dexterity constants resource file. Your macro will need to have steps in place to import the constants resource file.

4. You will also need scripts to drive a lot of the processes above, i.e., if you are launching Dexterity with a dictionary and a macro as a parameter to execute an automated task, this needs to be done by some task or step that supports this process. Anything scripting related, is preferable done with PowerShell since it has greater levels of automation over standard DOS batch files.

Now that you understand all the challenges, you can quickly guess that the best way to achieve this in Visual Studio Team Services is by setting up a Build Engine project with all the artifacts needed to automate the process.

The following shows the folder structure of a Build Engine project in VSTS:

Build Engine project structure

An explanation is as follows:

1. The DEX12, DEX14, and DEX16 folders contain full copies of the Dexterity IDE files, less things like samples, help files, and what's not. These are not needed as they will never be accessed.

2. The Logs folder will store any log produced by the macros being executed

3. The Macros folder includes all macros that we will need to compile and extract our code into chunk files.

If your macros refer to files in any of the folders in your Build Engine, these need to be relative to the root of structure of your project structure. In addition, it's easier to inject variables for things like the log paths, the dictionaries, version numbers, etc. It's considered best practice to not hard code any of these elements in your macro files to allow for reusability and customization. The following is an example of a chunking macro file:

---

# DEXVERSION=%DexMajorVersion%
Logging file '%LogPath%vDexUtilsMEP.log'
# ================================================================================
  MenuSelect title File entry 'Open Source Dictionary...' 
# ================================================================================
  FileOpen file '%ModulePath%/DYNMEP.dic' type 0 
# ================================================================================
ActivateWindow dictionary 'default'  form 'Main Menu' window 'DexUtils Toolbar' 
  MoveTo field 'Toolbar Utilities Button' item 0 
  ClickHit field 'Toolbar Utilities Button' item 6  # 'Extract' 
NewActiveWin dictionary 'default'  form Extractor window Extractor 
ActivateWindow dictionary 'default'  form Extractor window Extractor 
  ClickHit field '(L) Extract Button' 
# ================================================================================
  FileSave file '%ChunkDestination%/%VersionNumber%/MEP7156E.dic' 
# ================================================================================
NewActiveWin dictionary 'default'  form Extractor window Extractor 
NewActiveWin dictionary 'default'  form Extractor window Extractor 
  MenuSelect title File entry 'Close Source Dictionary' 
  MenuSelect title File entry 'Open Editable Dictionary...' 
# ================================================================================
  FileOpen file '%ChunkDestination%/%VersionNumber%/MEP7156E.dic' type 0 
# ================================================================================
ActivateWindow dictionary 'default'  form 'Main Menu' window 'DexUtils Toolbar' 
  ClickHit field 'Toolbar Utilities Button' item 9  # 'Auto-Chunk' 
NewActiveWin dictionary 'default'  form 'Auto Chunk' window 'Auto Chunk' 
ActivateWindow dictionary 'default'  form 'Auto Chunk' window 'Auto Chunk' 
  ClickHit field 'Lookup Button 3' 
# ================================================================================
  FileSave file '%ChunkDestination%/%VersionNumber%/MEP7156.cnk'
# ==================================================================================
  MoveTo field 'Dictionary Name' 
  TypeTo field 'Dictionary Name' , 'MEP7156.DIC'
  MoveTo field 'Dictionary Name' 
  MoveTo field '(L) Major Version' 
  TypeTo field '(L) Major Version' , '%DexMajorVersion%'
  MoveTo field '(L) Build Number' 
  MoveTo field '(L) Minor Version' 
  MoveTo field '(L) Build Number' 
  TypeTo field '(L) Build Number' , '%DexBuildVersion%'
  MoveTo field '(L) Major Version' 
  MoveTo field '(L) Build Number'  

---

You may be asking how these variables will be updated. Very simple: when we setup the Build process itself, we will create environment variables that can be injected into these macro files. You may also be asking how did these variables get there to begin with? Follow the steps outlined in Part 4 of the Dexterity Development Environment series to create your build macro and once you have recorded the macro, you can edit it to set up the place holders for environment variables.

4. The Resources folder contains things like constants resource files, clean dictionaries, and a Dex.ini file for the Dynamics GP versions that will be supported by your integrating Dexterity application.

5. The Scripts folder contains all the PowerShell scripts required to automate some of the tasks. One of such tasks is the ability to setup the dictionaries for the different products you will be building, as shown in this PowerShell script:

Param(
   [int] $VersionNumber,
   [int] $BuildNumber = "000",
   [int] $SubBuildNumber = "0",
   [string] $SingleModule = $null
)

$dictionary=''
$DexMajorVersion = 0
$localFolder = Get-Location #"$([System.IO.path]::GetPathRoot($(Get-Location)))" 

enum BuildType { 
    Source = 0
    Build = 1
}


switch ($VersionNumber) {
    2013 { 
   $dictionary = "$localFolder\resources\Dyn2013.dic"
   $DexMajorVersion = 12
  }
    2015 { 
   $dictionary = "$localFolder\resources\Dyn2015.dic"
   $DexMajorVersion = 14

  }
    2016 { 
   $dictionary = "$localFolder\resources\Dyn2016.dic"
   $DexMajorVersion = 16
  }

    default { 
   Write-Output "Invalid Version Number" 
   } 
}

$DexterityEXE = ".\Dex$DexMajorVersion\Dex.EXE"

$offset = "z_"  # Set blank for live mode.

if ($dictionary -ne ''){
    # Ensure the Destination folders exist.
    [System.Enum]::GetValues([BuildType]) | foreach { 
        $_compileModeName = $_
        mkdir "$localFolder\$($offset)$($_compileModeName)" -ErrorAction SilentlyContinue | Out-Null
        mkdir "$localFolder\$($offset)$($_compileModeName)\$VersionNumber" -ErrorAction SilentlyContinue | Out-Null
    }


    $fileSet = @()
    $modules = @('MICR', 'MICRJ', 'MMM', 'MEP', 'VPS')

    if ($modules -contains  $SingleModule)    
    { $modules = @($SingleModule.ToUpper()); Write-Host $modules -ForegroundColor Green }
    else { Write-Host "All Modules" -ForegroundColor Green }

    foreach($mod in $modules) {
        $workFolder = "$localFolder\$($offset)$($mod)\$($VersionNumber)"
        $dataFolder = "$workFolder\Data"
        mkdir $workFolder -ErrorAction SilentlyContinue | Out-Null
        mkdir $dataFolder -ErrorAction SilentlyContinue | Out-Null

        ## Copy the Dynamics Dictionary.
        $destination_file = "$workFolder\Dyn$($mod).dic"
     copy $($dictionary) $($destination_file)
        Set-ItemProperty $destination_file IsReadOnly -value $false
        Write-Host "$destination_file created." -Backgroundcolor Green -ForegroundColor Black

        # Copy Install_Code macro.
     $original_file = "$localFolder\Macros\Install_Code.mac"
     $destination_file =  "$workFolder\Install_Code.mac"
        $fileSet += , @($original_file, $destination_file, $mod, $_compileModeName)


        [System.Enum]::GetValues([BuildType]) | foreach { 
            $_compileModeName = $_
            $_compileMode = [int]$_

      ## Copy Constants.constant file to module-specific version and replace variables.
         $original_file = "$localFolder\resources\Constants.constants"
         $destination_file =  "$workFolder\Const_$_compileModeName.constants"
            $fileSet += , @($original_file, $destination_file, $mod, $_compileModeName)


            ## Copy the Macro - Twice: Once for Build and once for Source.
         $original_file = "$localFolder\Macros\Build_$($mod)_Source.mac"
         $destination_file =  "$workFolder\$($mod)_$($_compileModeName).mac"
            $fileSet += , @($original_file, $destination_file, $mod, $_compileModeName)

            ## Copy the Install_Constants.mac - Twice: Once for Build and once for Source.
         $original_file = "$localFolder\Macros\Install_Constants.mac"
         $destination_file =  "$workFolder\Install_Constants_$_compileModeName.mac"
            $fileSet += , @($original_file, $destination_file, $mod, $_compileModeName)
        }
        

  ## Copy Dex.INI file to module-specific version and replace variables.
  $original_file = "$localFolder\resources\Dex.ini"
  $destination_file = "$dataFolder\Dex.ini"
        $fileSet += , @($original_file, $destination_file, $mod, $compileMode)

        ## After this point, $MOD for MMM is MPP.
        if ($mod -eq 'MMM') { $mod = 'MPP' }
    }

    foreach($item in $fileSet)
    {
        $originating = $item[0]
        $destination = $item[1]
        $mod = $item[2]
        $_compileModeName = $item[3]
        $_compileMode = [int]$item[3]

        $workFolder = "$localFolder\$($offset)$($mod)\$($VersionNumber)"
        $chunkDestinationFolder = "$localFolder\$($offset)$($_compileModeName)"

        if ($_compileModeName -eq [BuildType]::Build.ToString())
        {
            $ChunkTypeComment = ""
            $_subBuildMessage = ""
        }
        else 
        {
            $ChunkTypeComment = "# -- Source -- " 
            $_subBuildMessage = "Build $($BuildNumber).$($SubBuildNumber) ($($BuildNumber).$($SubBuildNumber).$(Get-Date -format yyyyMMdd.hhmmss))"
        }


     (Get-Content $originating) | Foreach-Object {
      $_ -replace '%CompileMode%', "$_compileMode" `
         -replace '%CompileModeName%', "$_compileModeName" `
         -replace '%ChunkDestination%', "$chunkDestinationFolder" `
         -replace '%ChunkTypeConstant%', "$_compileMode" `
         -replace '%DexMajorVersion%', "$DexMajorVersion" `
         -replace '%DexBuildVersion%', "$BuildNumber" `
         -replace '%DexSubBuildMessage%', "$_subBuildMessage" `
         -replace '%ModulePath%', "$workFolder" `
               -replace '%Module%', "$mod" `
         -replace '%GenericPath%', "$localFolder\Generic\" `
         -replace '%TempPath%', "$localFolder\Temp\" `
         -replace '%LogPath%', "$localFolder\Logs\" `
         -replace '%OriginatingPath%', "$localFolder\Resources\" `
               -replace '%ChunkTypeComment%', "$ChunkTypeComment" `
         -replace '%VersionNumber%', "$VersionNumber"
        } | Set-Content $destination

        Write-Host "$($destination) created." -Backgroundcolor DarkRed -ForegroundColor Yellow
    }

    
    $DexterityEXE = ".\Dex$DexMajorVersion\Dex.EXE"
    $DexUtilsEXE = ".\Dex$DexMajorVersion\DexUtils.EXE"

    foreach($mod in $modules) {
        $workFolder = "$localFolder\z_$mod\$($VersionNumber)"
        $dictionary = "$workFolder\Dyn$($mod).dic"
        $LoadMacro =  "$workFolder\Install_Code.mac"

        Start-Process -FilePath $DexterityEXE -ArgumentList @($dictionary, $LoadMacro) -PassThru -Wait -Verbose
##        $process = (Start-Process -FilePath $DexterityEXE -ArgumentList @($dictionary, $LoadMacro) -PassThru -Wait)
##        Write-Host "$($mod) code loaded. Status: " $process.ExitCode
    }
}

6. The Source folder will contain the code retrieved from the Visual Studio Team Services repository. You will need a script to do this as well. The script will leverage the VSTS API to retrieve the code from the repo.

Param(
    [string]$SourceCodeFolder = "$/MICR/Base/2/2015B160", # Ex. 2015
    [string]$VSTSUser = "youraccount@somedomain.com",
    [string]$VSTSUserPAToken = "abcdefghijklmnopqrstuvwxyz0123456789",
    [switch]$TestStructure
)

$localFolder = Get-Location #"$([System.IO.path]::GetPathRoot($(Get-Location)))" 
$workFolder = "$localFolder\Generic\"

$scopePath_Escaped = [uri]::EscapeDataString($SourceCodeFolder) # Need to have this in 'escaped' form.

Write-Host "Pulling code from $($SourceCodeFolder) into $($workFolder)" -BackgroundColor DarkGreen

$recursion = 'Full' # OneLevel or Full
#$recursion = 'OneLevel' # or Full
 
# Base64-encodes the Personal Access Token (PAT) appropriately
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $VSTSUser,$VSTSUserPAToken)))
 
# Construct the REST URL to obtain the MetaData for the folders / files.
$uri = "https://yourcompany.visualstudio.com/DefaultCollection/_apis/tfvc/items?scopePath=$($scopePath_Escaped)&recursionLevel=$($recursion)&api-version=2.2"

# Invoke the REST call and capture the results
$result = $null
$result = Invoke-RestMethod -Uri $uri -Method Get -ContentType "application/json" -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}

 
# This call returns the METADATA for the folder and files. No File contents are included.
if ($result.count -eq 0)
{
     throw "Unable to locate code at $($SourceCodeFolder)"
}

$scopePathLength = $SourceCodeFolder.Length + 1 # +1 to eliminate an odd prefixed '\' character.

# =======================
# Create folder structure.
# =======================
for($index=0; $index -lt $result.count; $index = $index + 1)
{
    if ($result.value[$index].isFolder -eq $true)
    {
        #Strip the VSTS path off of the folder, and prefix with the local folder.
        $_subPath = $result.value[$index].path
        if ($_subPath.Length -ge $scopePathLength)
        {
            #Strip the VSTS path off of the folder
            $_subPath = $result.value[$index].path.Substring($scopePathLength)

            #prefix with the local folder.
            #MGB: replace the forward slashes in the remaining VSTS path with backslashes
            $_subPath = "$($workFolder)$($_subPath)" -replace "/", "\"

            if ((Test-Path $_subPath) -ne $true)
            {
                New-Item -Force -ItemType directory -Path $_subPath | Out-Null
                Write-Host $_subPath -BackgroundColor red
            }
            else
            {
                Write-Host "$($_subPath)`t$($result.value[$index].path)" -BackgroundColor Green -ForegroundColor Black
            }
        }
    }
}

# ==============
# Retrieve Files 
# -TestStructure flag will show all folders/files that will be retrieved.
# ==============
for($index=0; $index -lt $result.count; $index = $index + 1)
{
    if ($result.value[$index].isFolder -ne $true)
    {
        #Strip the VSTS path off of the folder, and prefix with the local folder.
        $_subPath = $result.value[$index].path
        if ($_subPath.Length -ge $scopePathLength)
        {
            #Strip the VSTS path off of the folder
            $_subPath = $result.value[$index].path.Substring($scopePathLength)

            #prefix with the local folder.
            #MGB: replace the forward slashes in the remaining VSTS path with backslashes
            $_subPath = "$($workFolder)$($_subPath)" -replace "/", "\"

            ## Retrieve the file text.
            if ($TestStructure -eq $true)
            {
                $fileresult = $result.value[$index].url
            }
            else
            {
                $fileresult = Invoke-RestMethod -Uri $result.value[$index].url -Method Get -ContentType "application/json" -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}
            }

            New-Item -Force -ItemType file -Path $_subPath -Value $fileresult | Out-Null
            Write-Host $_subPath -BackgroundColor Green
        }
    }
}

You can use the Visual Studio IDE to setup a project on VSTS and add the folders and files needed for your Build Engine project.

In summary, setting up a Build Engine project involves thinking about all the elements required to produce your chunk file: IDE, macros, and scripts that will drive the process. The complexity of each script will depend on the number of points you want to automate, variables and constants you want to inject, and certainly the number of chunks you need to produce for each version of Dynamics GP.

Tomorrow, I will explain the details involved with setting up the actual Build process with the parts.

Until next post!

MG.-
Mariano Gomez, MVP
IntellPartners, LLC
http://www.IntellPartners.com/