Working with Windows Azure Storage –Download Blob files – PowerShell

 

How to download blob files from the storage container.

 

1 Import-Module "C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Azure.psd1"
2 Import-AzurePublishSettingsFile C:\Azure\Azure_credentials_file.publishsettings
3 Get-AzureStorageAccount | select StorageAccountName
4 Set-AzureSubscription -SubscriptionName "SubscriptionName" -CurrentStorageAccountName "dbproxazurestorage"
5 Get-AzureStorageContainer  "containerstorage"  | Get-AzureStorageBlob | Get-AzureStorageBlobContent –Destination "C:\Azure\AzureStorageTableStore\blob_files_azurestorage"

 

 

 

Working with Windows Azure Storage –Azure Table Storage – PowerShell Part – 1

Azure Table Storage is a fault-tolerant, ISO 27001 certified NoSQL key-value store. Azure Table Storage can be useful for applications that must store large amounts of non-relational data, and need additional structure for that data. Tables offer key-based access to un-schematized data at a low cost for applications with simplified data-access patterns. While Azure Table Storage stores structured data without schemas, it does not provide any way to represent relationships between the data.

Despite some notable differences, Microsoft Azure SQL Database and Azure Table Storage are both highly available managed services with a 99.9% monthly SLA.

 

 

There are couple of free tools to work with Azure Storage (Blob / Tables / Queues). The one i most of the time use is Azure Storage Explorer. There is not GUI available on

https://portal.azure.com or https://manage.windowsazure.com

 

Download the tool and configure it by clicking Add Account (You need storage account,storage account key)

 

By Default 2 keys columns where created which cannot be deleted or modified.More columns can be added  by click View Button

 

In this example i have customer deta feed files in .csv format which i am uploading it to Azure Table from my local computer. You can also use the blob storage or VPC storage for faster upload. As i am uploading it over internet the upload will be slow depending upon the link.

Click on the View button

select the file, pass the partition Key Name (Unique record) and RowKey (rowguid).Click  upload

 

 

 

As the data is uploading you can click refresh to see how the data is uploading.

 

We can use the monitoring section in the https://manage.windowsazure.com

 

 

Check Azure Tables using PowerShell

 

How pricing / Billing is calculated for Blob / Storage Table / Queue read this article carefully.http://blogs.msdn.com/b/windowsazurestorage/archive/2010/07/09/understanding-windows-azure-storage-billing-bandwidth-transactions-and-capacity.aspx

Working with Windows Azure Storage –Remove Storage Blob- PowerShell Part – 6

 

Create the New-AzureStorageContext  and use the below command to delete the files.

 

—-Creating Azure Storage Context—

Import-Module “C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Azure.psd1”
Import-AzurePublishSettingsFile C:\Azure\Azure_credentials_file.publishsettings
$subscriptionName = “Subscription Name”
$storageAccountName = “Storage Account Name”
Select-AzureSubscription $subscriptionName
$storageaccountkey = get-azurestoragekey $storageAccountName | %{$_.Primary}
$destContext = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageaccou…

Get-AzureStorageBlob -Container containerstorage -Context $destContext

 

Get-AzureStorageBlob -Container containername -Context $destContext

 

Working with Windows Azure Storage –Remove Storage Account- PowerShell Part – 5

 

Continuation from Part – 4 . How to Remove Azure Storage Account (Empty/In Use)

Remove-AzureStorageAccount and pass the Storage Account Name when prompted

or

Remove-AzureStorageAccount  -StorageAccountName storageaccountname

 

Working with Windows Azure Storage –Remove Storage Container – PowerShell Part – 4

Continuation  from Part –3. Removing Azure Storage Container. While working with Azure i make sure i do not create anything which is not required/if testing, drop it once done. Azure services are paid and creating things which are not required may be chargeable (You will be billed for the services you use)

 

Lets see how can we drop the storage containers attached to the Storage Account.

 

Create the Storage Account context as explained in part-3.

Get the list of Azure Container attached to a storage Account

Get-AzureStorageContainer -Context $destContext

 

Drop the container

Remove-AzureStorageContainer -Name testcontainer1 -Context $destContext

 

 

Working with Windows Azure Storage –Create Storage Container – PowerShell Part – 3

Continuation  from Part –2. A storage account can contain zero or more containers. A container contains properties, metadata, and zero or more blobs. A blob is any single entity comprised of binary data, properties, and metadata.

 

Lets create a storage container under storage account we created in Part-2

Steps

1. Open PowerShell window and import the Azure PowerShell Module

Import-Module “C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ ServiceManagement\Azure\Azure.psd1”

1. Import the Credentials Publishing settings

Import-AzurePublishSettingsFile C:\Azure\Azure_credentials_file.publishsettings

1. Get details of Azure Storage Accounts

Get-AzureStorageAccount | SELECT StorageAccountName,Location,EndPoints |Format-Custom

1. Create the container under any of the account listed in step3. To create the container you require subscription Name,StorageAccountName,storageaccountkey to create AzureStorageContext.

Get-AzureSubscription PS C:\> $subscriptionName = “subscription name” PS C:\> $storageAccountName = “storage account name” PS C:\> Select-AzureSubscription $subscriptionName PS C:\> $storageaccountkey = get-azurestoragekey $storageAccountName | %{$_.Primary} PS C:\> $destContext = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageaccountkey PS C:\> New-AzureStorageContainer testcontainer1 -Permission Off -Context $destContext

Working with Windows Azure Storage –New Storage Account- PowerShell Part – 2

 

Automating Windows Azure Storage Account creation.. Using this you can create/drop multiple storage account for a customer or organization

Couple of things you need to keep in mind is “No use of CAPITAL letters”   for the [Storage Account Name] or [Label], the script will fail as you can see in the below images.

Know the Azure Module Location Handy if you are using Azure PowerShell Console

C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure

Know the AzurePublishSettingFile location

Import-AzurePublishSettingsFile C:\Azure\Azure_credentials_file.publishsettings

 

 

Steps to Create Azure Storage Account

1. Open PowerShell Window and Import PowerShell ModuleImport-Module “C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Azure.psd1”
   
2. Import the Azure Publish Setting File
   
   Import-AzurePublishSettingsFile C:\Azure\Azure_credentials_file.publishsettings
   
3. Create the Azure Storage Account
   
   New-AzureStorageAccount -StorageAccountName “dbproxpstest” -Location “Southeast Asia” -Label “psstorage”

   
   
   

Working with Windows Azure Storage –– PowerShell Part – 1

 

This post explains working with Windows Azure Storage using PowerShell.

 

Import the Azure Module

Import-Module “C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ServiceManagement\Azure\Azure.psd1”

Import AzurePulishingSettingFile (How to get AzurePublishingSettingFile)

Import-AzurePublishSettingsFile C:\Azure\Azure_credentials_file.publishsettings

There is currently a lease on the blob and no lease ID was specified for request ID {Azure Storage Blob }

Summary : The subjected issue occurred on while taking SQL Server Database backup on Windows Azure Storage.

 

The issue is very common , this normally comes when the URL back fails to complete, while the backup files is created on azure storage (backup failure do not drop the file @the storage).Now when the user tries to delete the backup(blob) files using storage portal or Azure Storage Explore  it pop’s up the below message.

 

Error deleting blob ‘AdventureWorks20121_FULL.bak’

Back to progress operations

There is currently a lease on the blob and no lease ID was specified in the request. RequestId:ba9c4274-0001-00c7-0351-516134000000 Time:2015-03-05T16:31:31.8959787Z

T-SQL Backup failure error

Error:

Msg 3202, Level 16, State 1, Line 14
Write on “https://dbproxazurestorage.blob.core.windows.net/containerstorage/AdventureWorks20121_FULL.bak” failed: Backup to URL received an exception from the remote endpoint. Exception Message: The remote server returned an error: (500) Internal Server Error.
Msg 3271, Level 16, State 1, Line 14
A nonrecoverable I/O error occurred on file “https://dbproxazurestorage.blob.core.windows.net/containerstorage/AdventureWorks20121_FULL.bak:” Error could not be gathered from Remote Endpoint.
Msg 3013, Level 16, State 1, Line 14
BACKUP DATABASE is terminating abnormally.

 

 

Investigation / Troubleshooting

When backing up to or restoring from Windows Azure storage, SQL Server acquires an infinite lease in order to lock exclusive access to the blob. When the backup or restore process is successfully completed, the lease is released. If a backup or restore fails, the backup process attempts to clean up any invalid blob. However, if the backup fails due to prolonged or sustained network connectivity failure, the backup process may not be able gain access to the blob and the blob may remain orphaned. This means that the blob cannot be written to or deleted until the lease is released.

 

Resolution

I have to admit that Azure portal  is not friendly enough to provide the exact issue or error.Though it give little information but that is not enough to resolve the issue.To get to the root of the issue download Azure Storage Explore  and connect to you Azure Storage.

 

 

select the backup file and click view from the tool bar. It will popup the property page of the backup.In this you will see the lease duration is set to Infinite and Lease  status is locked (Highlighted in Red).

To delete the files the lease has to be unlocked.This is was a real pain , after spending quite some time and couple of searches i came across this informative Microsoft article. The article has a power shell script to unlock the lease of the blob files. you can find the script below.

If there are blobs with locked leases, you should see the following messages:

Breaking Leases

The lease on <URL of the Blob> is a restore lease: You will see this message only if you have a blob with a restore lease that is still active.

The lease on <URL of the Blob> is not a restore lease Breaking lease on <URL of the Bob>.

 

After the lease is broken , you can easily delete the files from portal or Azure Storage Explorer.

 

 

PS Script:

[string]$storageAccount=’dbproxazurestorage’
[string]$storageKey=’Provide your storage key’
[string]$blobContainer=’containerstorage’
[string]$storageAssemblyPath=’C:\Program Files\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQL\Binn\Microsoft.WindowsAzure.Storage.dll’

# Well known Restore Lease ID
$restoreLeaseId = “8D2256EAD49C35D”

# Load the storage assembly without locking the file for the duration of the PowerShell session
$bytes = [System.IO.File]::ReadAllBytes($storageAssemblyPath)
[System.Reflection.Assembly]::Load($bytes)

$cred = New-Object ‘Microsoft.WindowsAzure.Storage.Auth.StorageCredentials’ $storageAccount, $storageKey

$client = New-Object ‘Microsoft.WindowsAzure.Storage.Blob.CloudBlobClient’ “https://$storageAccount.blob.core.windows.net”, $cred

$container = $client.GetContainerReference($blobContainer)

#list all the blobs
$allBlobs = $container.ListBlobs()

$lockedBlobs = @()
# filter blobs that are have Lease Status as “locked”
foreach($blob in $allBlobs)
{
$blobProperties = $blob.Properties
if($blobProperties.LeaseStatus -eq “Locked”)
{
$lockedBlobs += $blob

}
}

if ($lockedBlobs.Count -eq 0)
{
Write-Host ” There are no blobs with locked lease status”
}
if($lockedBlobs.Count -gt 0)
{
write-host “Breaking leases”
foreach($blob in $lockedBlobs )
{
try
{
$blob.AcquireLease($null, $restoreLeaseId, $null, $null, $null)
Write-Host “The lease on $($blob.Uri) is a restore lease”
}
catch [Microsoft.WindowsAzure.Storage.StorageException]
{
if($_.Exception.RequestInformation.HttpStatusCode -eq 409)
{
Write-Host “The lease on $($blob.Uri) is not a restore lease”
}
}

Write-Host “Breaking lease on $($blob.Uri)”
$blob.BreakLease($(New-TimeSpan), $null, $null, $null) | Out-Null
}
}