Tuesday, October 17, 2017

Copy all Azure Tables from one storage account to another

As a follow up to my previous post about copying blobs, here's how you can copy tables. This is even more rough around the edges than copying blobs. Firstly, you cannot just copy the table; you have to export it, then import it. Further, this cannot be done all in azure. When exporting, even if you use the command to export to blob container, it will download locally, then upload to the blob container so this can be quite a bit slower and incur additional charges so make sure you know what you're getting into.

The Script


cd 'C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy'

$sourceStorageAccountName = "SOURCE_STORAGE_ACCOUNT_NAME"
$sourceStorageAccountKey = "SOURCE_STORAGE_ACCOUNT_ACCESS_KEY"

$destStorageAccountName = "DESTINATION_STORAGE_ACCOUNT_NAME"
$destStorageAccountKey = "DESTINATION_STORAGE_ACCOUNT_ACCESS_KEY"
$destTemporaryContainerName = $(((Get-Date -Format o) -Replace '[^a-zA-Z0-9]','').ToLower())

$sourceStorageAccount = New-AzureStorageContext -StorageAccountName $sourceStorageAccountName -StorageAccountKey $sourceStorageAccountKey
$destStorageAccount = New-AzureStorageContext -StorageAccountName $destStorageAccountName -StorageAccountKey $destStorageAccountKey

$tables = Get-AzureStorageTable -Context $sourceStorageAccount
foreach($table in $tables) {
 Write-Host "Copying source table $($table.Name) from $($sourceStorageAccountName) to temporary storage container $($destTemporaryContainerName) on $($destStorageAccountName)"
 .\AzCopy.exe /Source:https://$sourceStorageAccountName.table.core.windows.net/$($table.Name)/ /Dest:https://$destStorageAccountName.blob.core.windows.net/$destTemporaryContainerName/ /SourceKey:$sourceStorageAccountKey /Destkey:$destStorageAccountKey /Manifest:"$($table.Name).manifest"
 Write-Host "Finished copying source table $($table.Name) from $($sourceStorageAccountName) to temporary storage container $($destTemporaryContainerName) on $($destStorageAccountName)"
 
 Write-Host "Importing data into destination table $($table.Name) from temporary storage container $($destTemporaryContainerName) on $($destStorageAccountName)"
 .\AzCopy.exe /Source:https://$destStorageAccountName.blob.core.windows.net/$destTemporaryContainerName/ /Dest:https://$destStorageAccountName.table.core.windows.net/$($table.Name)/ /SourceKey:$destStorageAccountKey /DestKey:$destStorageAccountKey /Manifest:"$($table.Name).manifest" /EntityOperation:"InsertOrReplace"
 Write-Host "Finished importing data into destination table $($table.Name) from temporary storage container $($destTemporaryContainerName) on $($destStorageAccountName)"
}

Write-Host "Deleting temporary storage container $($destTemporaryContainerName) on $($destStorageAccountName)"
Remove-AzureStorageContainer -Context $destStorageAccount -Name $destTemporaryContainerName -Force
Write-Host "Finished deleting temporary storage container $($destTemporaryContainerName) on $($destStorageAccountName)"

2 comments:

  1. Great! Worked like a charm. Thanks :)

    ReplyDelete
  2. If you are still looking for solutions to this problem, please look at a package I released on NuGet AzureTableUtilities -- https://www.nuget.org/packages/TheByteStuff.AzureTableUtilities/

    You can copy a table to a file or blob file, restore a table from a file or blob file as well as copy to another table on the same account or a different account. Filtering on PartitionKey and Timestamp is also available.

    I also created a reference command line code base and put it on GitHub -- https://github.com/TheByteStuff/AzureTableBackupRestore which allows this to be executed from a command line or Docker container.

    Let me know if this doesn't quite match what you want, and I'll see if I can enhance the functionality.

    ReplyDelete